INTERACTIVE RELATIVE ORIENTATION BETWEEN TERRESTRIAL IMAGES AND AIRBORNE LASER SCANNING DATA

INTERACTIVE RELATIVE ORIENTATION BETWEEN TERRESTRIAL IMAGES AND AIRBORNE LASER SCANNING DATA Peti Rönnholm *, Hannu Hyyppä, Pettei Pöntinen, Henik Haggén Institute of Photogammety and Remote Sensing, Helsinki Univesity of Technology, PL, 5 Espoo, Finland peti.onnholm@hut.fi, hannu.hyyppa@hut.fi, pettei.pontinen@hut.fi, henik.haggen@hut.fi KE WORDS: Inteactive Relative Oientation, Aibone Lase Scanning, Teestial Panoamic Images ABSTRACT: This pape descibes diect inteactive elative oientation method between teestial digital images and lase scanne data. Taditionally, the oientation between images and 3D data equies inteactive wok and accuate efeence points o visible featues. Lase scanning data does not easily allow detection of beaklines o cone points. Due to the huge amount of data, howeve, it is possible with human intelligence and inteactive oientation method to peceive the entity and to egiste teestial images and lase data to each othe. The pape descibes the basic tools fo inteactive oientation and contains discussion about the applicability of the method. An example fom a test site is depicted in ode to demonstate inteactive oientation in a pactical situation. 1. INTRODUCTION Aibone lase scanning technology has challenged the taditional measuements in uban and ual envionments in Euope since 1996. Lase scanne suvey povides a cloud of 3D points with accuacy almost equivalent to taditional land suveys. Lase scanning and photogammety can be seen as complementay methods. The lase is based on anging and is quick aibone method to poduce vetically oiented image data i.e. elevations. It woks well in the aeas that ae difficult fo steeoscopic photogammety. Howeve, lase is sensitive to scanning couption because of its geometic extapolation, wheeas photogammety is obust in its geomety. The focus of the eseach in the field of lase scanning has moved apidly fom digital teain models towads city models, extaction and modeling of buildings, segmentation and filteing, and integation of lase scanning data and digital images. Deficiencies of lase data include the lack of visual object infomation, limited geometic accuacy, the modeate esolution and the lack of edundancy needed fo the quality contol. Optical images can help to veify and fill these gaps. Howeve, the integation o data fusion of these two data sets is in its ealy stages. The use of optical images togethe with aibone lase scanning is tackled mainly concening aeial images. Howeve, digital teestial images have been used with teestial lase scanning (Pfeffei and Rottensteine, ) in inteios modelling. The othe focus aeas in lase scanning ae the quality of lase scanning (Hyyppä and Hyyppä, ; Ahokas et al., ; Huising and Gomes Peeia, 1998), and the accuacy and the eos in lase scanning (Schenk, ; Buman, 2). The high quality digital teain models with egula gids o TIN ae insufficient in the needs of accuate design o planning puposes. The knowledge ove the pecise position of beaklines is equied (Bügelmann, 2; Soininen, ). Taditional photogammetic methods to solve image oientation using 3D infomation ae insufficient, because the extaction of accuate cone points o vectos fom the lase scanning data is difficult. With inteactive oientation, it is, howeve, possible to solve elative oientation, because human intelligence can easily fit the entity. The aim of this pape is to pesent a novel algoithm fo diect elative oientation between teestial close-ange images and lase scanning data. The inteactive oientation method is based on visual intepetation of lase scanning data supeimposed on images. The ability of the method is veified with Real-Time Kinematic GPS obsevations. The pape also discusses about the potential of combined use of aibone lase data and teestial digital images based on panoamic mosaics. 2. MATERIAL 2.1 Panoamic Teestial Images Composed digital panoamic images ae useful fo teestial photogammetic measuements offeing ulta-wide viewing angle and high image esolution. In ode to ceate consistent panoamic images, the systematic lens distotions must be eliminated fom the oiginal images (Pöntinen, 2). Teestial images wee chosen, because they offe impoved complementay infomation to lase scanne data. Almost the same viewing diection of aeial images and aibone lase scanning can be advantageous fo some puposes, but fom teestial images it is feasible to visualize tagets that ae uneachable fom both of these methods. In addition, teestial point of view gives an excellent oppotunity to see, how lase altimety behaves on the gound. Two teestial panoamic images, establishing a steeo pai, wee ceated fom the Helsinki Univesity of Technology (HUT) campus aea in Otaniemi. The panoamic images, with the sizes of (9185 x 4939) (Figue 1) and (1729 x 5558) pixels, wee composed both fom 7 oiginal images. The image captuing, in autumn, was caied out using special panoamic platfom fo the digital camea (Figue 2) (Kukko,

; Pöntinen, ) making it possible to otate the camea aound its pojection cente. The camea used was Olympus C14L, which was calibated in HUT photogammetic 3D test field. Details of ceating panoamic images fom concentic image sequences ae pesented e.g. in Haggén et al. (1998) and Pöntinen (2) and system calibation in Hatley (1994). points wee measued using Real-Time Kinematic (RTK) GPS eceive. Hoizontal accuacy of the RTK was veified in Bilke and Kaatinen () to be about.15 m and vetical accuacy.2 m. Altogethe, ove 15 RTK-points wee measued, 2/3 of them wee gound points and 1/3 wee tees, buildings, walls, lamps etc. The camea locations wee also measued. 3. INTERACTIVE ORIENTATION METHOD 3.1 Rotating Image Fo image oientation, the 3D-otation matix must be solved. In the case of inteactive oientation, it is necessay to be able to contol the elements of the otation matix in a sensible manne. A 3D-otation matix fom the camea coodinate system to the gound coodinate system can be witten ( HUT / P. Pöntinen) Figue 1. Teestial panoamic image, the size of (9185 x 4939) fom Otaniemi test-aea, is composed fom 7 oiginal images. R =, (1a) which is usually explained with sequential otations (omega, phi, kappa) aound camea coodinate axes (x, y, z) when ( HUT / P. Pöntinen, A. Kukko) Figue 2. HUT- camea otation platfom enables concentic imaging of the hemisphee. 2.2 Lase Scanning Acquisitions Lase scanning campaign was caied out in August with TopEye and TopTea in Otaniemi. The test site consists of 2 flight stips, ca 2.5 km in length. Otaniemi teain can be chaacteized as flat and the size of the test site is appoximately 2.5 km x 3.5 km. The TopEye data set is acquied fom the altitude of 2 m esulting in aveage pulse ate between 2-5 points pe m 2. The swath width was about m. The accuacy of digital elevation models (DEM) in Otaniemi and the accuacy of taget models in Otaniemi wee studied in Hyyppä and Hyyppä (). Height eos and accuacy of lase scanning in vaying teain conditions and diffeences in heights wee calculated in Otaniemi aea. The aveage vetical shift between efeence data and lase data was.8 m, and the standad deviation of diffeences was about.5 m. 2.3 Gound Tuth Data Gound tuth data fom Otaniemi test site was collected with the Finnish Geodetic Institute. In ode to be able to veify the accuacy of lase points in Otaniemi, coodinates of efeence = cos ϕ cos κ = cos ϕ sin κ = sin ϕ = cos ω sin κ + sin ω sin ϕ cos κ = cos ω cos κ sin ω sin ϕ sin κ = sin ω cos ϕ = sin ω sin κ cos ω sin ϕ cos κ = sin ω cos κ + cos ω sin ϕ sin κ = cos ω cos ϕ o with azimuth, tilt and swing (Figue 3) when = cos α cos κ sin α cos ν sin κ = cos α sin κ + cos α cos ν cos κ = sin α sin ν = sin α cos κ + cos α cos ν sin κ = sin α sin κ + cos α cos ν cos κ = cos α sin ν = sin ν sin κ = sin ν cos κ = cos ν (1b) (1c) All the otations in equations 1b and 1c ae defined to be positive to the clock-wise diection. Being analogical to manual camea opeations, azimuth, tilt and swing system tends to be easie to undestand and to pedict its behaviou with teestial images.

z κ O y ϕ ω x X Z shifted = n X + Z oiginal. (4) ν c The paamete n in equations 2, 3 and 4 defines amount of shifts. The coefficients (... ) ae elements of 3D otation matix (eq. 1). image y z fowad nadi up backwad Figue 3. Typical otations: aound the axes of the camea coodinate system (ω, ϕ, κ) o azimuth, tilt and swing (α, ν, κ). 3.2 Shifts of Camea Location α noth down left image ight x Camea location must be found o known to solve inteactively the coect 3D otations of a camea. Shifts in the diection of eithe gound coodinate system axes o image coodinate system axes can be applied. Both methods have thei own benefits. If the image plane is not paallel to eithe gound coodinate system axis, it might be difficult to pedict how X- o -shifts effect to an image oientation. The Z-diection is easie to undestand and to handle, howeve. If shifts along the camea coodinate system ae used, it is easie to ealize, how changes affect to the view. Shifts can be named: left, ight, up, down, fowad and backwad (Figue 4). When a camea has tilt o swing, any movement affect to moe than one gound coodinate diections. The diections of camea coodinate system axes can be found diectly fom a 3D otation matix. The fist ow of a 3D otation matix fom a camea to the gound (eq. 1) is a unit vecto pojection of the viewing vecto of the camea onto the x-axis of the camea, the second ow is a unit vecto pojection of the viewing vecto onto the y-axis and the thid ow is a unit vecto pojection of the viewing vecto onto the z-axis. Shifts of the pojection cente of the camea, o any 3D point (X,, Z) in gound coodinates, to the diection of the x-axis of the camea coodinate system can be witten (Gube, 2) X Z shifted = n X + Z oiginal. (2) Shifts, in gound coodinates, to the diection of the y-axis of the camea coodinate system can be witten (Gube, 2) X Z shifted = n X + Z oiginal. (3) Shifts, in gound coodinates, to the diection of the z-axis of the camea coodinate system can be witten (Gube, 2) Figue 4. When shifts ae made along camea coodinate axes, movements can be called: left, ight, up, down, fowad and backwad. 3.3 Backpojection of Lase Scanning Data When changes ae applied to the otations o to the location of the camea, the lase scanning data must be supeimposed to the image with cuent oientation paametes in ode to see the esult. The backpojection of 3D point onto an image is calculated by the collineaity equations: ( X X x = c ( X X ( X X y = c ( X X ) + ) + ) + ) + ( ) + ( ) + ( ) + ( ) + ( Z Z ) ( Z Z ), (5) ( Z Z ) ( Z Z ) whee c is a camea constant, a point ( X,, Z ) is the pojection cente of a camea, (X,, Z) is a 3D gound point and tems (... ) ae elements of 3D otation matix (see eq. 1). The oigin of an image coodinate system is assumed to be at the pinciple point of an image. 3.4 Lase Data Selection When an aibone lase scanning data is examined fom a teestial point of view, it might be confusing to distinguish distances fom backpojected lase data, because tagets close and fa fom the camea ae mixing togethe. A good appoach to make side view easie to undestand is to use colo-coding by the distance (Fosman, ). Fo an inteactive elative oientation, thee is no need to use all lase scanning data available. Only some clealy visible featues fom diffeent sides of an image can be chosen to achieve a small set of data compaed to the complete data. Backpojecting all the data onto an image can be quite time consuming. Using educed set of lase scanning data inteactive oientation

becomes moe sensible, because the visual espond of change of oientation paametes is faste. The pulse density on the gound is dependent on flying height, pulse ate, scanning method and if the flying stips ae upon the each othe. 4. RESULTS AND DISCUSSION Two teestial panoamic images fom the test aea in Otaniemi wee oiented inteactively using a selected set of lase data. An oveall image (Figue 5) exhibits, how the backpojected lase scanning data fits the fist image afte the inteactive oientation. Moe detailed example fom the same image is pesented in Figue 6. Afte the inteactive oientation, the obtained camea locations wee compaed to the locations measued by RTK GPS (Table 1). Compaison eveals that the diffeences in heights between these two souces wee mino. The esult was as expected, since the imaging geomety is stong fo this diection. Figue 7 shows how a 6 cm shift in camea height is clealy visible even fom a consticted image. ( HUT / P. Rönnholm) Figue 5. Oveview of inteactively oiented teestial panoamic image and back pojected lase scanning data. Coodinate diffeence Image 1 (m) Image 2 (m) -.5 -.9 X -.141 -.78 Z. -.38 Table 1. Compaison between RTK efeence measuements and inteactively oiented camea locations. Planimetic eos wee bigge than height eos. This esult was also expected fom the imaging geomety. If the camea is moved fowad o backwad (see Figue 4) the effect on the image is much smalle compaed to effects when moving left, ight, up o down. The fine-tuning is enabled only if the viewing angle of the image is wide and the image esolution is high. These ae the main easons, why panoamic images ae useful to achieve a highe planimetic accuacy. ( HUT / P. Rönnholm) Figue 6. Resolution of panoamic image allows detailed views. One poblem with test images used was that thee wee only planimetic featues close to camea. This was not the best case fo an inteactive oientation. Howeve, if the oientations wee pefomed with the camea locations measued with RTK, it was not possible to fit exactly the lase points on the image. Theefoe, thee ae pobably some planimetic shifts between RTK and lase scanning data, and only pat of planimetic eos can be explained with eos of inteactive oientation. Oveall, the esults wee distinctly pomising. Using the developed novel method, we can get good estimates of shifts between lase scanning data and equied gound coodinate system. Instead of measuing dozens o hundeds of contol points fom the lase scanning aea, it is possible to measue only the camea location and then make inteactively elative oientation between an image and lase scanne data. Solving camea location with taditional exteio oientation of an image using gound contol points is also possible, of couse, if it is not easy to measue the camea location diectly o if the coect oientation angles need to be solved as well. The quality of inteactive elative oientation depends mainly on the quality of lase scanning data. Futhemoe, the oientation is the moe obust the moe thee ae samples pe m 2 available. ( HUT / P. Rönnholm) Figue 7. Shift of 6 cm in camea height is clealy visible fom the image. Expeiences with test images aised some ideas, how to incease planimetic accuacy with good imaging planning. If two images with 9 degee viewing diections ae used, the affect of the weake geometical conditions along the viewing diection of a camea is bypassed. In addition, if elatively oiented image pais ae used instead of single images, the method should be much moe obust. With steeo image pai it is also possible to investigate the behavio of a lase scanning data in 3D.

Woking with inteactive oientation method equies pope tools. The biggest poblem is that afte any shifts to camea location the otations must be changed befoe the final esult of changes is clealy visible. This makes it moe difficult to pedict, how much the values should be changed at time. The use of an ancho point is a valuable tool to patially solve this poblem. Howeve, if a lase point is used as an ancho point, it is usually necessay to change it duing the inteactive oientation. In some cases, some pe-knowledge about the camea location o otations is known and the inteactive oientation method is fast and easy to handle. Table 2 accumulates some typical cases with inteactive oientation method and descibes the level of use fiendliness. A ule of thumb with inteactive oientation is that it is usually elatively fast and easy to find a coase oientation, but the fine-tuning equies caefulness and takes longe time. Popeties: Shifts along gound coodinate system axes Shifts along image coodinate system axes Shifts along image coodinate system axes + ancho point(s) Known camea location Known camea otations Usefulness: Slow, difficult to handle Faste, but still difficult to handle Much faste, easie to handle Fast and easy Fast and easy Table 1. Applicability of the inteactive oientation method in diffeent cases. 5. CONCLUSIONS We have pesented an inteactive elative oientation method that is feasible fo vesatile applications. Since 1998 we have used this method with known 3D points, vecto data, mobile imaging (Töönen et al., ), foested aeas (Jokinen et al., ) and both teestial and aibone lase scanne data. With accuate vecto data o 3D points, thee ae moe efficient and accuate methods to solve image oientations, but inteactive oientation can sometimes be faste and it deals obustly with data including seveal coase eos. With lase scanning data the situation is totally diffeent, because the data does not eveal easily accuate beaklines o cone points. Howeve, the human intelligence is able to adjust this difficult data souce quite easily with teestial images, if the scene contains enough identifiable featues. The accuacy of detecting diffeences in heights is high, if teestial images ae used with aibone lase scanning data. The planimetic accuacy is not as good, because of the weake imaging geomety. Howeve, if seveal panoamic images with diffeent viewing diections ae used, also the planimetic accuacy can be adequate to many puposes. With backpojecting a lase scanning data onto the oiented image it is easie to undestand the behavio of lase scanning. In addition, elative oientation assists to veify quickly, if thee ae some intenal eos, discontinues o gaps within lase data o if some changes have occued in the taget. Inteactive elative oientation enables to opeate without any contol points o digital teain model, if necessay. This could be advantageous in some aeas, if it is difficult to get efeence measuements. Inteactive oientation method is ou fist, easy and valuable step towads moe sophisticated methods fo elative oientation of aibone lase scanning data and teestial digital images. It is expected that combined use of lase scanning data and digital images will bing exta values to mapping and intepetation. Thee has been some discussion whethe lase scanning will displace the taditional photogammety. Both methods have some advantages and disadvantages and it would be gogeous to use the best possible popeties fom both of them. We believe that combining these two methods will actually incease the use of the photogammety in the futue. On the othe hand, the lase scanning will also benefit fom this pogess. 6. REFERENCES Ahokas, E., H. Hyyppä, J. Hyyppä, and H. Kaatinen,. Analysing the effects elated to the accuacy of lase scanning fo digital elevation and taget Models, Poceedings of Easel, nd EARSeL Symposium & Geneal Assembly, June 4-6,, Pague, Czech Republic. Bilke, M. and H. Kaatinen,. The Quality of Real-Time Kinematic (RTK) GPS Positioning. Repots of the Finnish Geodetic Institute :1, Kikkonummi. Bügelmann, R, 2. Automatic beakline detection fom aibonelase ange data. Intenational Achives of Photogammety and Remote Sensing. Vol XXXIII, Pat B3. Amstedam 2. Buman, H.. Lase Stip Adjustment fo Data Calibation and Veification. p. A-67 ff. ISPRS Commission III, Vol. 34, Pat 3B Photogammetic compute vision, Gaz,. Gube, D., 2. The Mathematics of the 3D Rotation Matix, Pesented at the Xteme Game Developes Confeence, Septembe 3-Octobe 1, 2, Santa Claa, Califonia, http://www.makegames.com/3dotation/ (accessed 8 May ). Fosman, P,. Thee dimensional localization and mapping of static envionments by means of mobile peception. Helsnki Univesity of Technology. Automation technology laboatoy. Seies A: Reseach Repots No.. p. 145. Haggén, H., P. Pöntinen and J. Mononen, 1998, Cocentic image captue fo photogammetic tiangulation and mapping and fo panoamic visualization. IS&T/SPIE's th Annual Symposium on Electonic Imaging: Science and Technology, to 29 Januay 1999, San Jose, Califonia USA, Poc. SPIE 3641, p.17-, 1998. Hatley, R., 1994. Self-Calibation fom multiple views with a Rotating Camea. Lectue Notes in Compute Science, Vol. 8, Jan-Olof Eklund (Ed.), Compute Vision - ECCV '94, Spinge-Velag Belin Heidelbeg 1994. Huising E.J. and Gomes Peeia, L.M., 1998, Eos and accuacy estimates of lase data acquied by vaious lase scanning systems fo topogaphic applications. ISPRS Jounal

of Photogammety and Remote Sensing. 53 (1998) pp. 245 261. Hyyppä J. and H. Hyyppä,. Lase Scanning eseach in Finland. Quality of lase scanning. Laseskanning och digitala bilde - idag och imogon - mak, hus, ledninga, täd och annan vegetation i 3D. Stockholm.1.. Jokinen, O., U. Pyysalo, P. Pöntinen, P. Rönnholm,. Detemination of coesponding tunks in a pai of teestial images and aibone lase scanne data. GeoInfomatics,. Vol. 6, 3. Kukko, A.,. Digitaalikamean asemointi pallopanoaamajalustaan (Panoamic Rotation Platfom fo Mounting of Digital Camea, in finnish), The Institute of Photogammety and Remote Sensing, HUT, ESPOO. 2 p. Pfeiffe, N. and G. Rottensteine,. The Riegl lase scanne fo the suvey of the inteios of Schönbunn palace. Fifth Confeence on Optical 3-D Measuement Techniques. Vienna. 4.-6.1.. Pöntinen, P., 2. On the ceation of panoamic images fom image sequences. XIXth Congess of the Intenational Society of Photogammety and Remote Sensing, Amstedam, 17.-.7.2. Amstedam 2, ISPRS, pp. 635-641. Pöntinen, P.,. Camea Calibation By Rotation. Intenational Society fo Photogammety and Remote Sensing - ISPRS Commission V, Septembe 3-7., Cofu, Geece. Pp. 585-589. Soininen A.,. New possibilities with lase scanning. Laseskanning och digitala bilde - idag och imogon - mak, hus, ledninga, täd och annan vegetation i 3D. Stockholm.1.. Töönen, J. (ed.), NAVI-PAM Poject Goup,. Pesonal Navigation Sevice Achitectue: Suvey of Backgound and Standads, VTT Infomation Technology, Espoo. 7. ACKNOWLEDGEMENTS The authos ae gateful to the Academy of Finland (poject numbes 76463, 53 and 276) and the Foundation of Jenny and Antti Wihui fo financial suppot.