LINE-BASED REFERENCING BETWEEN IMAGES AND LASER SCANNER DATA FOR IMAGE-BASED POINT CLOUD INTERPRETATION IN A CAD-ENVIRONMENT

Transcription

1 LINE-BASED REFERENCING BETWEEN IMAGES AND LASER SCANNER DATA FOR IMAGE-BASED POINT CLOUD INTERPRETATION IN A CAD-ENVIRONMENT N. Meiehold a, *, A. Bienet a, A. Schmich b a Technical Univesity of Desden, Institute of Photogammety and Remote Sensing, 6 Desden, Gemany - (Nadine.Meiehold, Anne.Bienet@tu-desden.de b kubit GmbH, Fiedlestaße 36, 37 Desden, Gemany - amin.schmich@kubit.de Commission V, WG V/3 - Teestial Lase Scanning KEY WORDS: Close ange photogammety, teestial lase scanning, data fusion, image oientation, line photogammety ABSTRACT: Photogammetic 3D object econstuction can be obtained fom images o lase scanne data. Because of thei complementay chaacteistics, these data ae suitable fo a combined intepetation. In this context, a new option fo 3D object econstuction appeas. 3D geometies can be obtained by monoplotting-like pocedues, mapping in monocula images and deiving depth infomation fom the lase scanne data. Fo this pupose, a coect geometic efeencing of the data is equied. The efeencing pimaily contains the detemination of the exteio oientation of a single image towads a lase scanne point cloud. In achitectual applications, linea featues can often be extacted easie than points. Hence, the pape descibes two methods fo image oientation based on staight line featues. Both appoaches ae based on the collineaity equations. The fist one uses coespondences between image points and 3D lines extacted fom the point cloud, wheeas the second appoach uses linea featues in image space and point cloud. These methods wee tested with two diffeent data sets and compaed to a classical photo esection using points. Fo the fist data set, a test field with a lage numbe of tagets, the esults wee also compaed to the esults of a bundle block adjustment. The second data set is a building facade, whee natual featues athe than signalised points wee the basis of image oientation. The esults show a diffeent accuacy potential fo the thee methods. As expected, the best esults wee obtained with the point-based photo esection, followed by the line-based method with the point-to-line coespondence. But it is also shown that using lines instead of discete points may be a valuable option fo the oientation of single digital images to lase scanne 3D point clouds.. INTRODUCTION The softwae poduct PointCloud of the company kubit GmbH in Desden is a softwae module fo visualisation and intepetation of lage point clouds in AutoCAD. In addition to inteactive point cloud pocessing by using the full CAD functionality, a (semi-automatic data modelling via plane fitting and intesecting these planes is possible. Futhemoe, a combined intepetation of digital images and teestial lase scanne data in a monoplotting-like pocedue is integated. This means that elements such as points o lines measued in an image can be pojected into the point cloud. This pocedue makes use of the advantages of both kinds of data: the high geometic esolution and the high visual quality of images as well as the accuate and eliable 3D infomation of lase scanne data. The peequisite fo the integated use of images and point clouds fo measuement and intepetation pocesses is a coect geometic efeencing of the data. Hence, the subject of this pape is the geometic efeencing of teestial lase scanne data and photos taken independently fom the lase scanne. The aim is to detemine the exteio oientation of a single image by measuing coesponding point and/o staight line featues in the image and a point cloud of the same scene. Because many uses (fo example achaeologists, achitects, monument consevatos will take the images with digital amateu cameas, a simultaneous calibation of the camea should be possible by detemining the inteio oientation and the lens distotion fo each image. Ealy investigations fo line photogammety deive fom the scope of aeial photogammety, e.g. (Finstewalde, 94. Since then, a lage numbe of publications have appeaed to the topic of line photogammety, e.g. (Habib et. al., 999; Patias et. al., 995; Tommaselli & Tozzi, 996; Hemken & Luhmann,. Schenk (4 distinguishes between two line-based appoaches: the collineaity and the coplanaity appoach. While the fist one is based on the collineaity equations, the second one is based on the assumption that the image line, the object line and the pespective cente ae coplana. Van den Heuvel (3 detemines the exteio oientation of an image by using pependicula coplana staight lines. Thee line intesection points and the pespective cente fom a tetahedon. Via distance popotions of abitaily cone points obtained by a volume appoach, the exteio oientation can be calculated. The two line-based methods fo image oientation which will be descibed in this pape belong to the goup of collineaity appoaches. They epesent a stepwise tansition fom the classical single photo esection using contol points to a spatial esection using staight line featues descibed by line equations. The diffeent appoaches wee implemented and tested with eal datasets. The esults will be shown and evaluated in the face of applicability fo the efeencing of image data and lase scanne data. * Coesponding autho

2 . POINT-BASED IMAGE ORIENTATION This section shall only be egaded as a shot intoduction to the topic of image oientation. The oientation of a single image pimaily contains the detemination of the exteio oientation. That includes the oientation angles and the position of the pespective cente in elation to the object efeence system. The detemination of the oientation paametes is ealised indiectly by measuing imaged object featues in the photo. If these featues ae contol points with known coodinates in the image and in the object efeence system given by the lase scanne data, the image oientation can be ealised by a classical single photo esection. The applied collineaity model (Eq. descibes the pojection of an object point onto a twodimensional senso of a camea in consideation of the exteio oientation (X, Y, Z, ω, ϕ, κ, the inteio oientation (c, x, y and imaging eos (dx, dy. x = x 3 y = y 3 ( X X + ( Y Y + 3 ( Z Z ( X X + 3 ( Y Y + 33 ( Z Z ( X X + ( Y Y + 3 ( Z Z ( X X + ( Y Y + ( Z Z dx + dy ( extacted by the intesection of planes fitted into a segmented point cloud, e.g. (Vosselman et. al., 4. Using lines athe than points may also be advantageous if pats of the object ae occluded. Moeove, the automation of extacting lines fom 3D point clouds may be consideed to be easie than the extaction of discete points. These ae good easons fo switching to methods woking with staight lines, which ae pesent in many man-made objects like building facades. 3. 3D line epesentation Thee ae diffeent methods fo descibing staight lines in Euclidean 3D space. Hee, a desciption with fou paametes is chosen, which was published by Robets (988 and esumed and vaied by Schenk (4. Apat fom its cleaness, this line epesentation has two main advantages. Fistly, it is unique and fee of singulaities. Consequently, no special cases have to be consideed. Secondly, the numbe of paametes is equal to the degee of feedom of a 3D line. Theefoe, this desciption is paticulaly suitable fo photo intesection and bundle adjustment applications. The two positional paametes (X s, Y s and the oientation paametes (α, θ ae visualised in Fig.. whee ij = elements of otation matix x, y = image coodinates X, Y, Z = object coodinates The imaging eos shall hee be limited to the adial distotion, whose coection can be descibed by the following equation based on Bown (97. dx = x A = x dy = y A ( + A ( + A3 ( ( + A ( + A ( + y whee = senso specific paamete (zeo-distotion adius x, y = image coodinates elated to the pinciple point At least thee eo-fee non-collinea contol points ae equied to calculate the paametes of exteio oientation. The use of moe points leads to a non-linea adjustment poblem, which is solved iteatively minimising the sum of the squaed deviations between the modelled and the obseved image featues. If a sufficient numbe of suitably distibuted contol points is available, the paametes of the inteio oientation and of the lens distotion can be estimated simultaneously. This is mainly useful fo photos taken with an amateu camea because the pincipal point and the pinciple distance may have to be egaded as vaiant fo all images (Maas, LINE-BASED IMAGE ORIENTATION Concening the geometic efeencing of images and lase scanne point clouds, the necessity of measuing discete points is disadvantageous. As a consequence of the sub-sampling chaacteistics of lase scanne data, distinctive points ae not always well defined in 3D point clouds. This effect gows with deceasing scan esolution and/o inceasing distance fom the object. By using line-based appoaches, it is not necessay to measue identical points in the image and the lase scanne point cloud. Instead, lines in images may be extacted by image pocessing techniques, and lines in 3D point clouds may be 3 ( Figue. Paamete of 3D line epesentation The azimuth α and the zenith angle θ can be deduced fom the spheical coodinates of the diection vecto d of the line (Fig., left. To get a unique line epesentation the anges of the angles have to be limited to < α < 36 and < θ < 9. This coesponds with the estiction to the hemisphee of the positive Z-axis. These angles ae used to collimate the Z-axis and the line by otating the coodinate system. The following matix descibes the otation about the Z-axis by angle α and the subsequent otation about the new Y-axis by angle θ. R αθ cos α cos θ = sin α cos α sin θ sin α cos θ cos α sin α sin θ sin θ cos θ The esulting X - and Y -axis eside on a plane which passes though the oigin and is pependicula to the line. The two positional paametes esult fom the intesection of the line with this plane (Fig., ight. The intesection point matches the point of shotest distance between the line and the oigin. X s p = Ys = R t αθ p The otation of any point p of the line to the new coodinate system (Eq. 4 leads to the same planimetic coodinates but to diffeent Z-coodinates. Thus, paamete t coesponds to the (3 (4

3 line paamete of the point-oientation epesentation of a line. The line equation can be deduced by invesion of Eq. 4: X s cos α cos θ Ys sin α + t cos α sin θ T = (5 p R αθ p = X s sin α cos θ + Ys cos α + t sin α sin θ X s sin θ + t cos θ 3. The point-to-line appoach A line appoach using a point-to-line coespondence was published by Kubik (99. His aim was to detemine the seven tansfomation paametes fo the absolute oientation of images by eplacing the contol points in the efeence system by contol lines. Fo aeial tiangulation, Schenk (4 implemented a block adjustment with extended collineaity equations using staight lines. This publication povides the basis fo the fist method fo single photo oientation descibed in the following. Fo the efeencing of photos and lase scanne point clouds, point-to-line coespondence means the definition of staight lines in the point cloud and the measuement of points on the coesponding lines in the image. At this, the 3D lines ae descibed by 4 paametes as pesented in section 3.. Fo a coect oientation, the pojection ay fom the pespective cente though the image points should intesect the coesponding object lines (Fig.. Figue. Pinciple of point-to-line appoach This point-to-line coespondence can be descibed mathematically by extending the collineaity equations. At this, the object coodinates (X, Y, Z ae eplaced with the espective ow of the line equation in Eq. 5: x = x y = y 3 3 ( X X + ( Y Y + 3 ( Z Z ( X X + 3 ( Y Y + 33 ( Z Z ( X X + ( Y Y + 3 ( Z Z ( X X + ( Y Y + ( Z Z X X s cosα cosθ Ys sinα + t cosα sinθ with Y = X s sinα cosθ + Ys cosα + t sinα sinθ Z X s sinθ + t cosθ dx + dy Like in the pue point-based method, the paametes of the exteio oientation (X, Y, Z, ω, ϕ, κ ae estimated in an iteative pocess, whee the esiduals of the obseved image coodinates ae minimised. Additionally the line paamete t has to be estimated fo evey image point. Measuing two image points pe object line, at least thee known non-paallel object lines ae equied fo detemination of the exteio oientation *. The paametes of the inteio oientation (x, y, c as well as the paametes of the adial distotion (A, A, A 3 can be * 6+6 unknowns; 6x obsevations (6 calculated fom at least six known spatially distibuted object lines (e.g. in planes with diffeent levels of depth with two coesponding image points. Fo heightening the edundancy and consequently impoving the accuacy of the adjustment, the numbe of image points pe object line can be inceased. 3.3 The line-to-line appoach This section deals with a line-based appoach, which was published by Schwemann (995. Hee, the oientation of a single image is based on the definition of staight object lines in the point cloud and on the measuement of coesponding image lines. The equations of the image lines ae defined as: g : y = m x + n o x = m y + n (7 whee m, m = slope n, n = intesection with appopiate axis Schwemann (995 used this type of line equations fo the desciption of the object lines as well. Because this epesentation is only applicable fo staight lines with small slopes, thee ae many special cases that must be consideed with espect to appopiate axes. To avoid these case diffeentiations, the method was vaied by applying the 4- paamete line equations pesented in section 3.. The basic pinciple of this line-to-line appoach is the modification of the collineaity equations in such a way that the image line paametes (m, n ae dependent on the paametes of the object line (X s, Y s, α, θ and the paametes of the exteio and inteio oientation (pinciple point and pinciple distance. In the couse of the modification, the unknown line paamete t was eliminated. The esulting equations ae shown below: VW VW m = U W U W VW V W UV U V n = y x + c U W U W U W U W whee U = cos α sin θ + sin α sin θ + V = cos α sin θ + sin α sin θ + W = 3 cos α sin θ + 3 sin α sin θ + U = V = W = with A = X s cos α cos θ Ys sin α B = X s sin α cos θ + Ys cos α C = X sin θ ( A X + ( B Y + 3 ( C Z ( A X + ( B Y + 3 ( C Z ( A X + ( B Y + ( C Z s cos θ cos θ cos θ In the special case of a vetical image line, the invese elations of Eq. have to be consideed as obsevations in the nonlinea adjustment pocess. n m = and n = ( m m The detemination of the exteio oientation of an image can be obtained fom at least thee staight lines defined in the lase scanne point cloud. The model does not compise any (8 (9

4 coection tems fo lens distotion, because the coection model in Eq. is not applicable. Hence, eithe thee has to be deived anothe distotion model which is dependent on the image line paametes (m, n, o the lens distotion has to be detemined in a pe-calibation befoe the image oientation pocess. An integated oientation pocedue might be developed using elements fom plumb-line calibation (Lema & Cabelles, 7. In the case of pe-coection, the measuement of the image lines would be caied out in the esampled image. 4. COMPARISON OF THE METHODS 4. Inteaction in the CAD-envionment The pesented image oientation methods wee implemented in the softwae poduct PointCloud of the company kubit GmbH. The typical wokflow of image oientation in PointCloud is shown in the following Figue. distotion esampled image is visualised and stoed by the PointCloud application using a customised AutoCAD item. Now the use can choose between measuing inteactively in the oiented image and/o in the point cloud itself by making use of the appopiate PointCloud tools. 4. Datasets The intoduced methods fo image oientation wee tested with two diffeent data sets. The fist data set is a spatial test field with an extension of x 3 mete. It consists of 7 staight lines (8 hoizontal, vetical, 9 tansvesal, which ae each maked with fou tagets. The lase scanne data wee acquied with a Riegl LMS Z-4i. Additionally, 3 convegent images wee obtained of the test field. In the couse of analysing the images in AICON 3D studio, the 3D coodinates of the signalised line points wee detemined. The exteio oientations of the images calculated in the bundle block adjustment act as efeee fo the single image oientations in PointCloud. The object is not vey elevant fo pactical application, but it allows fo a thoough evaluation of the accuacy potential due to the lage numbe of well defined tagets. Figue 3. Wokflow of image oientation in PointCloud The fist step is the impot of a teestial lase scanne point cloud and an image of the same scene. Fom that moment all use inteactions concening the image oientation ae caied out in the CAD-envionment. These inteactions include the measuement of point o line featues in the image and in the 3D point cloud as well as the appoximate setting of camea position and viewing diection to the object. One big advantage of the CAD is the visualisation which enables the use to contol his measuements and the esults diectly. Afte efeencing the data, the oiented and fo lens Figue 4. Image and lase scanne point cloud of data set Nymphenbad The Nymphenbad, a pat of the histoical building Zwinge in the city of Desden, is object of the second data set. The southeasten facade was scanned with a point spacing of mm (depending on the scanning distance and a total of 3 million 3D points (Fig. 4. The images of both data sets wee acquied with a 6 Mpix mio eflex camea with 7.8 µm pixel size. 4.3 Results bundle block adjustment aveaged intenal standad deviations of estimated paametes method s s X [mm] s Y [mm] s Z [mm] s ω [ ] s ϕ [ ] s κ [ ].5 pixel esection with points.4 pixel esection with lines (point-to-line esection with lines (line-to-line method.75 pixel [ ] Table. Means of standad deviations of estimated paametes of exteio oientation aveaged absolute diffeences of estimated paametes to bundle block adjustment X [mm] Y [mm] Z [mm] ω [ ] ϕ [ ] κ [ ] esection with points esection with lines (point-to-line esection with lines (line-to-line Table. Compaison of exteio oientations obtained by single image esections and bundle block adjustment

5 4.3. Data set test field : The measuement of the signalised line points in the lase scanne data was not possible because of the eto-eflecting chaacteistics of the signalised points which caused incoect distance obsevations of the lase scanne. Theefoe, object coodinates obtained fom the bundle block adjustment wee used fo analysing this data set. The exteio oientations of thee out of the 3 convegent images wee calculated wheeat the inteio camea oientation was deived fom the block adjustment. Fist, the signalised line points wee used as contol points fo a conventional pointbased spatial esection. Then the esection was pefomed on the basis of staight line featues, each defined by two signalised line points, using the two methods as outlined befoe. The esulting oientation paametes wee compaed to the exteio oientation obtained fom the bundle block adjustment. The absolute diffeences aveaged ove thee images ae shown in Tab.. The empiical standad deviations of unit weight and of the unknowns (aveaged ove thee images ae summaised in Tab.. Fom these tables, a degadation of pecision between the thee esection methods is obvious. As expected, the best esults could be obtained with the conventional point-based photo esection. Fo instance, its standad deviation is with.4 Pixel twice as good as when efeencing with the point-to-line appoach. The line-to-line appoach shows pooe intenal pecision and much lage diffeences to the check points. One eason fo this is in the neglection of lens distotion in the oientation model. RMS X [mm] RMS Y [mm] RMS Z [mm] point-to-point point-to-line line-to-line esection method Table 3. RMS of diffeences between contol points and points obtained by intesection In a next step, the object coodinates of 33 points wee calculated by spatial intesection using the thee images each oiented with the thee methods. A compaison to contol points is shown in Tab. 3. This test is not elevant fo the actual application, but it gives some infomation on the quality of the oientation paametes. It shows that the pecision of the point- esection method to-line method is almost as good as the pecision of the conventional esection, while the line-to-line method pefoms -3 times wose Data set Nymphenbad : This data set of a histoical building facade was chosen fo testing the methods unde ealistic conditions. In the case of point-based efeencing, the oientation of the image is based on natual points (e.g. distinctive cones, cossbas. Fo the line-based methods, 6 non-paallel staight lines with two points pe line wee used. Fig. 5 shows the oiented image ovelaid with the lase scanne point cloud. The featues wee measued in the image and the lase scanne data inteactively. Because of the unknown inteio oientation of the camea, it was detemined simultaneously togethe with the adial distotion paametes A and A (lens distotion not in line-to-line method. The compaison of the classical photo esection and the esection methods using linea featues ae shown in Tab. 4 and Tab. 5. Figue 5. With point-line-method oiented image in PointCloud ovelaid with the lase scanne point cloud In these tables the same degadation between the methods is appeaing. Fom the smalle standad deviations and the smalle diffeences, one can conclude that the point-to-line appoach yields bette oientation esults than the line-to-line esection method. Additionally, it is noticeable that the gap between the point-based esection and the point-to-line appoach has become smalle. intenal standad deviations of unknowns s sx [m] sy [m] sz [m] sω [ ] sϕ [ ] sκ [ ] sc [mm] sx [mm] sy [mm] point-to-point.5 pixel point-to-line.8 pixel line-to-line.66 [ ] Table 4. Standad deviations of image oientation by applying the thee esection methods esection method diffeences of estimated paametes to point-to-point esection method X [m] Y [m] Z [m] ω [ ] ϕ [ ] κ [ ] c [mm] x [mm] y [mm] point-to-line line-to-line Table 5. Compaison of oientation paametes obtained by point-based and line-based single photo esection

6 The almost identical esults of the point-based and the point-toline esection and the pooe pecision of all esection methods in compaison to the data set test field can pimaily be explained by less accuate object coodinates. With an inceasing distance of the lase scanne to the object (hee... 8 m, the point spacing inceases and the accuacy of 3D featues inteactively measued in the point cloud deceases. Theefoe, it can be expected that the point-to-line esection will pefom bette than the point-based algoithm if 3D line featues ae obtained fom (semi-automatic modelling (e.g. by intesecting of planes fitted into the point cloud athe than fom inteactive measuements. 5. CONCLUSIONS Using lines athe than discete points may be a valuable option fo the oientation of single digital images to lase scanne 3D point clouds. Two diffeent algoithms, a point-to-line and a line-to-line coespondence appoach, have been developed, implemented and tested. The esults show that, compaed to discete points, line featues ae equally well o even bette suited fo the efeencing between images and lase scanne point clouds. This fact is ponounced with an inceasing point spacing. Accuacy-wise, the point-to-line esection pefoms bette than the line-to-line appoach. In addition to a bette image oientation, the point-to-line method has anothe impotant advantage because of its coespondence between image points and object lines: staight 3D lines do not have to be imaged as staight lines. That means thee is no necessity fo detemination of lens distotion and esampling of the image befoe image oientation. Consequently, image featues can be measued in the oiginal image. If imaging eos of the optical system ae not consideed, appoaches based on staight lines in the image space ae only applicable fo an appoximate image oientation. While the point-to-line method pefoms bette in (semi- inteactive measuement, the line-to-line appoach may depict a bette option fo a fully automatic system, whee linea featues can be deived fully automatically in the point cloud and the image. This will be a majo issue of futue wok on the topic. ACKNOWLEDGEMENT The eseach wok is funded by esouces of the Euopean Fund of Regional Development (EFRE and by esouces of the State Saxony within the technology development poject CADbasiete bildgestützte Intepetation von Punktwolken. Hemken, H. & Luhmann, T.,. Mehbildoientieung mit geometischen Elementen. Abschlussbeicht Foschungsschwepunkt Raum-Roh-Boden, Teilpojekt 3, Wissenschaftliche Beicht de Fachhochschule Oldenbug/Ostfiesland/Wilhelmshaven, N.. Kubik, K., 99. Relative and absolute oientation based on linea featues. ISPRS Jounal of Photogammety and Remote Sensing, 46 (99, pp Lema, J. L. & Cabelles, M., 7. A eview and analyses of plumb-line calibation. The Photogammetic Recod, (8 (June 7, pp Maas, H.-G., 998. Ein Ansatz zu Selbstkalibieung von Kameas mit instabile innee Oientieung. Publikationen de Deutschen Gesellschaft fü Photogammetie, Fenekundung und Geoinfomation e.v., 8. DGPF-Jahestagung Munich, Vol. 7, pp Patias, P., Petsa, E. & Steilein, A., 995. Digital Line Photogammety. Eidgenössische Technische Hochschule Züich, Institut fü Geodäsie und Photogammetie, Beicht N. 5. Robets, K. S., 988. A New Repesentation fo Lines. IEEE Poceedings of Compute Vision and Patten Recognition, (988, pp Schenk, T., 4. Fom point-based to featue-based aeial tiangulation. ISPRS Jounal of Photogammety and Remote Sensing, 58 (4, pp Schwemann, R., 995. Geadengestützte Bildoientieung in de Nahbeeichsphotogammetie. Dissetation, Veöffentlichung des Geodätischen Instituts de Rheinisch-Westfälischen Technischen Hochschule Aachen, N 5. Tommaselli, A. M. G., & Tozzi, C. L., 996. A ecusive appoach to space esection using staight lines. Photogammetic Engineeing & Remote Sensing, Vol. 6, No., Januay 996, pp van den Heuvel, F. 3. Automation in Achitectual Photogammety. PhD thesis, Publication on Geodesy 54, NCG, Nethelands Geodetic Commission, Delft Vosselman, G., Gote, B.G.H., Sithole, G. & Rabbani, T., 4. Recognising stuctue in lase scanne point clouds. Intenational Achives of Photogammety, Remote Sensing and Spatial Infomation Sciences, Vol. 46, pat 8/W, Feibug, Gemany, Octobe 4-6, pp REFERENCES Bown, D.C., 97. Close-ange camea calibation. Photogammetic Engineeing, 37 (8, pp Finstewalde, S., 94. Die gemeinsame Koppelung deie Luftaufnahmen desselben Geländes. Sitzungsbeichte de Bayeischen Akademie de Wissenschaften, Mathematisch- Natuwissenschaftliche Abteilung, Sondeduck, pp Habib, A., Asmamaw, A., Kelley, D. & May, M., 999. Linea Featues in Photogammety. Depatment of Civil and Envionment Engineeing and Geodetic Science, The Ohio State Univesity. Depatment Repot No. 45.