COMBINATION OF DISTANCE DATA WITH HIGH RESOLUTION IMAGES

Transcription

1 COMBINATION OF DISTANCE DATA WITH HIGH RESOLUTION IMAGES R. Reulke a HU-Belin, Compute Vision, 199 Belin, Gemany- eulke@infomatik.hu-belin.de Commission V, WG V/5 KEY WORDS: Vision, Fusion, Camea, Sanne, DEM/DTM, Othoetifiation ABSTRACT: Typial devies fo aquiing distane data ae lase sannes. Combining these with highe esolution image data is state of the at. Reently, amea systems ae available poviding distane and image data without usage of mehanial pats. Two manufatues of suh amea systems with a typial esolution of 16 x 12 pixels ae CSEM Swiss Range and PMDTehnologies GmbH. This pape desibes a design, ombining a PMD and a highe esolution RGB amea. Fo eo-fee opeation, alibation of both ameas and alignment detemination between both systems is neessay. This way, a pefomane detemination of the PMD system is also possible. 1. INTRODUCTION AND MOTIVATION The fusion of high esolution amea with lase sanne data is a pomising appoah and allows the ombination of highesolution image and depth data. Appliation aeas ae indoo building modelling, ompute vision and doumentation of ultual heitage (Reulke, 25). Reent methods offe 3D animated videos o inteative 3D models fo 3D sene visualization. The aquisition of additional ange data is uial fo eating 3D models and diffeent altenatives ae povided in lose-ange photogammety. Fo modelling a 3D sene, image o distane based appoahes ae possible. Distane elated sene modelling based on lase sanne data has been used fo lose-ange photogammety (e.g., aquisition of building geometies) fo seveal yeas. An oveview of atual available lase sanne systems an be found in (Weh 25). Many poets equie high esolution tue-olou sufae textues. Thee ae seveal ombined systems whih povide the option of additional olou data aptuing. This is solved by mounting a amea nea the lase diode o by omplex mehanial mio systems. Most systems use additional ommeial CCD fame ameas. Reently small and ompat imaging systems with distane measuement apability ae available. The pesent atile desibes the ombination of a highe amea and a PMD-system. Two manufatues of suh amea systems with a typial esolution of 16 x 12 pixels ae CSEM Swiss Range and PMDTehnologies GmbH. The pape is oganized as follows: In the seond hapte the PMD system will be desibed and in the following the system will be analysed. The fouth hapte desibes the esults of fusion with highe esolution image data. 2.1 PMD - Piniple 2. PMD CAMERA A Photoni Mixe Devie (PMD) allows aptuing the omplete 3D sene (distane and gay value) in eal time. The appoah is equivalent to lase sanne. A summey of diffeent distane measue piniples fo lase sanne an be found in (Weh, 25). Thee ae diffeent anging piniples applied in teestial lase sanning. In teestial lase sanning anging is aied out eithe by tiangulation o by measuing the time of flight of the lase signal. Fo the time-of-flight measuements, shot pulses with high peak powe ae podued by lases. The tavelling time of a lase pulse is measued. Anothe possibility to detemine the tavelling time of a signal is ealized by measuing the phase diffeene between the tansmitted and eeived signal. In ontinuous wave (w) lase anging the lase intensity is modulated with a well defined funtion e.g. a sinusoidal o a squae wave signal. The modulation signal is epeated ontinuously. The time of flight of the signal is detemined by measuing the phase diffeene between the tansmitted and the eeived signal. The peiod time T Pd, the fequeny f o wavelength λ defines the maximum unambiguous ange un, whih is λ/2 fo two-way anging. To obtain 3D infomation of the entie sene, a modulated single lase beam would have to be sanned mehanially ove the sene. In PMD eah pixel an individually measue phase shift of the modulated light. This is done by using ontinuous squae wave modulation and measuing the phase delay fo eah pixel. A speial advantage of PMD is that the omplete mixing poess of the eleti and optial signal takes plae within eah pixel. Deteto pixel dimension 4 µm x 4 µm # pixel (v,h) 16 (h) x 12 (v) Optis adapte C-mount Modulation fequeny 2 MHz (7.5 m ambiguity inteval) FOV mm Wavelength 87 nm Intefae FieWie, Ethenet Weight 14g Table 1. Speifiation fo PMD [vision] 19k A modulated optial signal sent by a tansmitte (nea infaed light soue at λ = 85 nm), illuminates the sene to be measued. The efleted light entes the PMD senso, whih is also onneted to the modulation soue. The hages in the p- type semionduto, geneated by the photons, ae sepaated

2 inside the optially sensitive aea of the semionduto in elation to the efeene signal. The esulting output signal depends on the phase shift of the efleted optial signal (whih aies the desied 3D infomation) and the eletial modulation signals fo the illumination soue in the hip. Table 1 gives an oveview of the pefomane of the senso. 2.2 PMD Camea Calibation By using PMD measuements, distane infomation fo eah pixel is available. On the othe hand the spatial oodinates ae neessay. A elation fo deiving oodinate infomation fom this distane infomation an be deived. Figue 1 shows the geomety: Z PZ(X Y Z ) X z Y y y X Figue 1: Geometi elation fo PMD The oodinate measuement is elated to the poetion ente (PZ) of the PMD-amea. Theefoe the amea position and the view dietion, whih ae equivalent to the exteio oientation ae fixed in X =, Y =, Z = and ω=, ϕ=, κ=. Fo a pixel with the oodinates (x,y) a distane d(x,y) was measued. The obet oodinates (x,y,z ) in the PMD fame an be alulated as follows: x x P (x y ) D( P(x y z ) whee R ( ωϕκ,, ) X, Y, Z = Obet oodinates x, y = Image oodinates x, y, = Piniple point, foal length X, Y, Z = Poetion ente Matix fo otation into the image oodinate system A standad efeene fo inteio oientation is the Bown model (Bown, 1971) whih is implemented in the Austalis bundle blok adustment pogam (Fase, 2). The image oodinate oetion funtion in Austalis is the ommonly used 1-paamete model. The alibation paametes an be gouped as follows: Camea inteio oientation:, x, y Radial distotion paametes: k1, k2, k3 Deenting distotion paametes: p1, p2 Affinity, non-othogonality paametes: b1, b2 The oeted image oodinates (x o, y o ) an be alulated fom the measued oodinates (x meas, y meas ) using the fomulas x = xmeas x y = ymeas y = x + x xo = xmeas x + x= x+ x y = y y + y= y+ y o meas = x and y epesent the oetions due to the desibed effets. The PMD-amea was alibated with Austalis. The imaging mode of the PMD-amea gives poo image quality; theefoe an additional CCD-amea was used, to identify the image stutues in the PMD image. The amea is a DBK 41BF2 with 128 x 96 pixel, whih ealizes olou with Baye mosai (see web site fom TheImagingSoue). (3) z = d x y x = x y = y + x + y x + y x + y with 2 2 = d z (1) (, ) (x,y) ae the image oodinates in the PMD amea fame but the measuement is elated to ows and olumns. To detemine the image oodinates, piniple point, foal length and additional distotion effets have to be oeted. This is a typial poblem of amea alibation o inteio oientation detemination in photogammeti sense. The detemination of inteio and exteio oientation is based on the ollineaity equations. The ollineaity equation desibes the geomety between the sensos poetion ente, the oodinates of an obet and the image oodinates in elation to the piniple point. Figue 1 shows the geometi elation between poetion ente, obet points and image oodinates. To tansfom obet oodinates to image oodinates, the following elation an be used: ( X X ) + ( Y Y ) + ( Z Z ) ( X X ) + ( Y Y ) + ( Z Z ) x = x 31 X X + 32 Y Y + Z Z y = y 31 X X + 32 Y Y + Z Z (2) Figue 2 Geysale image of PMD-amea (left) and DBKamea (ight) Fo ompaison figue 2 shows the image of the PMD-amea and an image fom the DBK-amea. The stong signal hange in the PMD-image is elated to the own illumination system of the PMD-system. An additional spekle noise is also visible. The labels shown in the image ae also used fo alibation. The foal length of both systems is about 12 mm. In spite of the fat that the numbe of pixel diffes oughly by a fato of 8, the image of the obet is nealy the same. The eason is that the pixel size diffes by the same elation (4 µm pixel distane fo PMD to 5 µm fo the DBK). Paamete DBK PMD Senso_Size_H Senso_Size_V 96 12

3 Pixel_Size_H.5.4 Pixel_Size_V.5.4 C XP YP K e e-3 K e e-4 K e e-5 P e e-4 P e e-4 B e e-5 B e e-4 Table 2. Result of the alibation of PMD and DBK, paametes ae in Table 2 gives the esult of the alibation of both ameas. Patiulaly with egad to piniple point and distotion, signifiant influene is visible. The auay of the alibation an be measued with the standad deviation of the blok adustment sigma, whih is in the ange of 1/1 of a pixel distane fo DBK-amea and 1/4 of a pixel fo the PMD-amea. The eason fo the unsatisfatoy alibation esult of the PMD-amea is the auay of point detemination in the PMD-images. The known inteio oientation now allows the detemination of the exteio oientation fom known obet points e.g. with spatial esetion algoithm. Spatial Resetion is a widely used tool in photogammety fo nonlinea alulation of the exteio oientation fom moe then thee tagets with known oodinates. using the eigenveto assoiated with the smallest eigenvalue of the matix T B = A A (7) The matix A is fomulated suh that thee olumn ae [x i x, yi y, z i - z ] (Fobes, 1989). Mean and standad deviation of d i gives geneal infomation about epoduibility and auay of the PMD-devie. No z a b <d 2 i > Table 3. Auay of PMD distanes measuement with integation time of 1 ms. This esult shows a lage standad deviation of 75 mm. The eason is the lage diffeenes in efletion popeties of the makes. Following figue shows a sufae view of the distane data with an image ovelay. 3. INVESTIGATION OF PMD DATA 3.1 Repoduibility and auay alulation The detemination of the obet oodinates (x, y, z ) in the PMD fame an be obtained with the help of the known inteio oientation and by using equations (1). The auay detemination means the peision and the standad deviation fo distane measuement and was done with a plane sufae in font of the PMD-amea, as shown in figue 2. A plane is desibed by a point P (x,y,z ) and the dietion osines (a, b, ) of the nomal to the plane as follows (see also Luhmann, 23): a xi x b yi y zi z = + + (4) The distane fom any point Q(x i,y i, z i ) to a plane speified above is given by d = a x x + b y y + z z (5) i i i i The sum of squaes of distanes of eah point fom the plane is n F = d i= 1 2 i By using the optimal paametes (a,b,), F is minimised. The best-fit plane passes though (x,y,z ), whih is the aveage of the data points. The dietion osines (a,b,) an be obtained by (6) Figue 3: Distane eo due to hanging efletion popeties. A textuised sufae view. Seveal algoithms ae investigated to filte the distane eos due to stong efletion hanges and to find out optimal distane values. In a window of (2m+1, 2m+1) the efleted amplitudes ae analysed. If the amplitude is lage then a value, the distane value will be used fo futhe evaluation. The oeted distane at pixel (i, ) an be evaluated fom the emaining distane values by median o mean filte. d if a > (, ) [, ] i, i, di, = i m m if ai, o di, = di, The standad deviation alulated fom median is 17.5 mm and fom mean with 11 mm. Fo the following alulation the mean is used. This is equivalent to an auay of about 2.5 (fo an integation time of 1 ms) in elation to the obet distane. (8)

4 The integation time has a stong influene to the auay. Expeimental esults (see table 4) shows, that the integation time should be in a ange between 5 ms and 5 ms. Lowe values then 5 ms ae inauate beause of noise and poo efleted intensity. With highe integation (> 5 ms) time satuation effet an be seen. It seems, that distane measue depends fom the integation time. Fame integation time [ms] Distane standad deviation standad deviation/ distane [ ] Table 4. Auay of PMD distanes measuement with vaying integation time of 1 ms. (The distane is 4192 mm measued with a handheld lase angefinde.) The absolute auay was ompaed with a handheld lase angefinde and is in the ange of 5 mm. An auate ompaison with deived data fom photogammeti appoahes gives the same esult. Table 5 shows the esult fo diffeent distanes to a flat wall. Fame Measued distane PMD distane Standad deviation PMD PMD PMD Lab Lab x = R + x (9) As seen in figue 4 the tansfomation an be deived fom the exteio oientation of the PMD-amea, whih is equivalent to the tansfomation fom the amea oodinate system into the wold oodinate system: Z Lab Y Lab Figue 4: PMD and Lab-oodinate systems An additional otation of 18 aound the z-axis may be neessay, if the intenal oodinate system stats on the uppe left site. 4.2 Deivation of a DHM PMD x Lab Z PMD X Lab Y PMD X PMD Fo eah pixel in the PMD data set, the x, y and z oodinates an be deived. Beause of the lens distotion the deived DHM is not equidistant in x- and y-dietion. An additional esampling algoithm (gidding) of the (x, y, z) data on a egula gid is neessay. Diffeent appoahes and pogams ae available and implemented. Hee, an ENVI toolbox was used. Figue 9 shows the poessed PMD data fo DHM and the image. Table 5. Auay of distane measuement (integation time 5 ms) in ompaison with a handheld lase angefinde. Fo a lage distane than the maximum unambiguous ange un the obseved value is modulo un (=7.5 m). The standad deivation howeve is equivalent to 11.5 m and an be used as a tool to oveome ambiguities. The standad deviation fo PMD distane measuement is about.5-1, the epoduibility is less than.5% and the absolute auay is about 1%. 4. FUSION WITH HIGH RESOLUTION IMAGES Fo tests a small obet (at) of a size less than 1 m was used. 4.1 Co-alignment of diffeent ameas fames The detemination of the elative oientation of both ameas seems to be not auate enough beause of weak mehanial stability between the ameas. Theefoe obet oodinates deived fom PMD distane image ae tansfomed into the lab oodinate fame. Eah image fom othe ameas with known exteio and inteio oientation an be used fo ombination with PMD data.the elation between Lab and PMD oodinate system is: Figue 5. DHM (left) and image (ight) of the obet afte gidding The same obet was eoded with the highe esolution DBKamea. The mapping of the high esolution textue on the low esolution DHM is equivalent to the othophoto geneation. The known poedue (Luhmann, 23) an be ealized in thee steps:

5 Camea foal plane O p ( x,y ) digitale image p(, O p DGM Z(P) Z(P) otho image matix Figue 6. Othophoto geneation 1. Define otho-image size and pixel size. Fo an obet point, the equivalent Z-value an be detemined by intepolation of the gidded DHM data, whih is in the Lab oodinate system. 2. With the ollineaity equation the image point an be alulated. The exteio oientation of the amea has to be detemined befoe. 3. Using the inteio oientation model, the image point (in ows and olumns) an be detemined. The gey value fom this image point has to be used with the otho-image matix. The Bown model howeve was designed to alulate the oeted pixel fom measued pixel position. Theefoe an invese poedue is neessay, whih an iteatively alulate the unoeted pixel position fom oeted one. As a esult an otho-oeted image with highe esolution in ompaison to the DHM esolution an be deived. The following figue shows the otho-image and textue DHM. Refeenes fom Jounals: Figue 8. A textuized DHM Bown, D.C., Photometi Engineeing, pages , Vol. 37, No. 8. Fase, C. S., Edmundson, K. L. 2. Design and Implementation of a Computational Poessing System fo Off- Line Digital Close-Range Photogammety, ISPRS Jounal of Photogammety & Remote Sensing, 55(2): Reulke, R., Sheibe, K., Weh, A., 25. Integation of digital panoami amea and lase sanne data, Intenational Wokshop on Reoding, Modeling and Visualization of Cultual Heitage, Asona. Shwate R., A new eletooptial mixing oelation senso: Failities and Appliations of the Photoni Mixe Devie (PMD), Po. SPIE Vol. 31, Weh, A. 25. Lase Sanning and Its Potential to Suppot 3D Panoami Reoding, IAPRS, VOLUME XXXVI-5/W8, Editos: R. Reulke, U. Knaue Xu, Z., Shwate, R., Heinol, H., Buxbaum, B., Ringbek, T., Smat pixel - photoni mixe devie (PMD), New system onept of a 3D-imaging amea-on-a-hip. Refeenes fom Books: Figue 7. Result of the othopoto geneation. PMD image (left), etified DBK image (ight) Figue 8 shows the textuised DHM fom the obet. 5. CONCLUSION AND OUTLOOK The pesent pape investigates an imaging system with distane measuing apability. An appoah fo ombining distane and highe esolution image data was pesented. This is a peequisite e.g. fo eal time 3D imaging. The auay of the desibed poedue needs futhe investigation. Luhmann, T. Nahbeeihsphotogammetie, 2. Auflage, Wihmann Velag, 23 Refeenes fom Othe Liteatue: Fobes, A.B., Least-squae best-fit geometi elements. National Physial Laboatoy, Repot DITC 14/89, Teddington, United Kingdom. Refeenes fom websites: TheImagingSoue: ye/dbk41bf2/oveview/ Austalis: