BUNDLE BLOCK ADJUSTMENT OF OMNI-DIRECTIONAL IMAGES OBTAINED FROM A GROUND MOBILE MAPPING SYSTEM

Transcription

1 BUNDLE BLOK ADJUSTMENT OF OMNI-DIRETIONAL IMAGES OBTAINED FROM A GROUND MOBILE MAPPING SYSTEM T. Oh a K. hoi a I. Lee a * a Department of Geoinfomatics The Universit of Seoul 9 Jeonnong-dong Dongdeamun-gu Seoul Korea - (theidps shale iplee)@uos.ac.kr ommission I WG I/3 KEY WORDS: Omni-Directional amera Ground Mobile Mapping Sstem Bundle Block Adjustment GPS/INS Multi-sensors ABSTRAT: Regarding the increasing demands for high qualit spatial information man researchers have emphasized the need for multisensor/platform integration for more rapid and economical generation of such spatial data. Most sstems emploing a set of frame cameras ma have suffered from their small fields of view and poor base-distance ratio. These limitations can be significantl improved b emploing an omni-directional camera that is capable of acquiring images in ever direction. Using a GMMS integrated with this camera we can enlarge the mapping coverage and observe objects from a variet of directions and positions. Bundle Block Adjustment (BBA) is one of the existing georeferencing methods to determine the exterior orientation parameters of two or more images. In this stud b expanding the concept of the traditional BBA method we attempt to develop a mathematical model of BBA for omni-directional images with GPS/INS data and Ground ontrol Points (GPs). The proposed mathematical model includes two main parts; observation equations based on the collinearit equations newl derived for omni-directional images and stochastic constraints imposed from GPS/INS data and GPs. We also report the experimental results from the application of our proposed BBA to the real data obtained mainl in urban areas. With the different combinations of the constraints we applied four slightl different tpes of mathematical models. The tpe where onl GPs are used as the constraints provides the most accurate results less than 5 cm of RMSE in the estimated ground point coordinates. In future we plan to perform more sophisticated lens calibration for the omni-directional camera to improve the georeferencing accurac of omni-directional images. These georeferenced omnidirectional images can be effectivel utilized for cit modelling particularl autonomous texture mapping for realistic street view.. INTRODUTION As the demand for high qualit spatial information such as sophisticated 3D cit model are being increased man researchers have attempted multi-sensor/platform integration to obtain spatial data more rapidl and economicall. One of the most promising sstems to acquire spatial information along roads is a Ground Mobile Mapping Sstem (GMMS) usuall equipped with imaging cameras laser scanners and GPS/INS (Gontran 3; Vincent 5). Most sstems emploing a set of single frame cameras ma have problems due to limited Field Of View (FOV) and poor base-distance ratio. These problems can be substantiall solved b incorporating an omni-directional camera which can acquire an omni-directional field of view at a time without an movement or rotation of the camera (Aizawa 4). Using a GMMS with this camera we can observe objects from almost entire directions. Such capabilit allows us to generate more complete and realistic street view. In past most omni-directional cameras utilize spherical mirrors with a single detector inherentl retaining significant level of distortions (Silpa 5). Man researchers should attempt to remove or reduce such distortions (Beauchemin ). Later some recent omni-directional cameras such as Ladbug (Point Gre Research 8) are integrated with a set of single frame cameras and generate a large image showing scenes in all directions b stitching each image together. Such a camera can generall obtain high resolution omnidirectional images with small distortions which ma be also useful for 3D precision mapping of road-side objects. The mapping process requires accuratel georeferenced omnidirectional images. Georeferencing is a process to determine the exterior orientation parameters of images that is the position and attitude of the camera at the time of exposure for each image (hoi 9). One of the most popular georeferencing methods is Bundle Block Adjustment (BBA) where we can estimate exterior orientation parameters and ground point coordinates b adjusting the light bundles originating from image points. Traditionall the BBA method has been defined for a set of frame camera images and intensivel used for D or 3D mapping from these images. To appl this well-established method to omni-directional images rather than frame camera images we have to modif its mathematical models. Hence in this stud we attempt to develop the mathematical models of BBA for omni-directional images with GPS/INS data and GPs. The proposed mathematical model is derived through two steps. First we derive new collinearit equations for omnidirectional images representing the geometric relationships between a ground point and the corresponding image point on an omni-directional image. A set of observation equations based on these collinearit equations are then established. Second from GPS/INS data and GPs we formulate two kinds of stochastic constraints for BBA. Based on these two step derivation results we finall present four different models according to the selection of different unknowns and constraints. * orresponding author.

2 . Overview. METHODOLOGY As the main input the proposed BBA method requires tie points (TPs) manuall or automaticall selected from omnidirectional images. In addition to overcome the datum deficienc it also uses GPS/INS data and (or) GPs. With these inputs as like the traditional BBA methods the proposed method is also to estimate the EOPs of each omnidirectional image and GPs (Ground Points) corresponding to all the TPs. The mathematical models for the proposed BBA method consists of two main parts. The first one is the observation equations associated with TPs expressed as Eq. (). The second one is the stochastic constraints associated with GPS/INS data and GPs expressed as Eq. () and (3) respectivel. A. Observations: Y F Ξ E Ξ P + () B. onstraints: Z G Ξ E + () Z G Ξ P + e z (3) where Y observations derived from TPs Z constraints derived from GPS/INS data Z constraints derived from GPs the measurement errors associated with Y the measurement errors associated with Z e z the measurement errors associated with Z Ξ E unknowns for EOPs Ξ P unknowns for GPs The observation equations are based on the new collinearit equations associated with an omni-directional images rather than ordinar frame camera images. GPS/INS data and GPs are used as the stochastic constraints for EOP and GPs respectivel. More detail derivations in both parts are presented in the following sections.. Observation Equations based on ollinearit Equations The collinearit equations indicate the relationship between ground points and image points. An ordinar frame camera generates an image through the perspective projection of a 3D scene to a D focal plane. However an omni-directional camera generates an image through the perspective (central) projection to 3D spherical surface. Therefore we should derive new collinearit equations for omni-directional images with some modifications to the existing equations. Let assume that G P is the coordinate vector of a ground point expressed in a ground coordinate sstem (GS). This point can be expressed as G P in the camera coordinate sstem (S) using the coordinate transformation in Eq. (4). Here G O is the origin of S expressed in GS and G R is a rotation matrix from GS to S. The ground point is then projected to a spherical surface toward the projection center that is the origin of S as shown in Figure. The projected image point ρ can be expressed with a horizontal angle (α) and a vertical angle (β). These angles are expressed as Eq. (5) where P x P and Pz indicate the 3D coordinates of P. Figure. entral projection of GP to a spherical surface ρ α β atan P Px asin ( Pz/ P ) B combining Eq. (4) and (5) we can express the relationship between a GP ( G P) and the corresponding image point (ρ) on an omni-directional image with its EOP of G R and G O. The combined equations indicate the collinearit equations for omni-directional images. Based on the collinearit equations the observation equations for an image point can be expressed as Eq. (6) where σ is the variance of the image point measurement errors and I is b identit matrix. The equations actuall involves nine parameters the ground point G P ( G Px G P G Pz) the projection center G O (X Y Z ) and the rotational angles (ω ϕ κ ) for R G. ρ α β atan P Px asin ( Pz/ P ) (5) + e e ~( σ I ) (6) The nonlinear equations in Eq. (6) can be linearized into Eq. (7) using a Talor series. Here Ξ is the vector of nine parameters Ξ is its initial approximation; and f is the functions for the collinearit equations. ρ f(ξ ) + f/ Ξ ΞΞ (Ξ Ξ ) + e e~( σ I ) We can express the Jacobian matrix in Eq. (7) as F Ξ f Ξ f Ξ atan t t t Ξ asin t t t Ξ atan (t ) t t P asin (t ) t t P (7) P. (8) Ξ P G R G P G O (4) where t t P / P x / P P z (9)

3 The partial differentiation in Eq.(8) is expressed as P t t P atan t t asin t t P P x P z P + t t () P P x P x P x P z P P z P x + P P 3 P 3 P 3 () Finall P Ξ can be rewritten as Eq. () where Ξ is GP coordinates Ξ is the camera position parameters; and Ξ 3 is the camera attitude parameters as defined in Eq. (3-5). P P Ξ Ξ P Ξ P () Ξ 3 Ξ X Y Z T G P (3) Ξ X Y Z T G O (4) Ξ 3 ω ϕ κ T (5) The differentiation of P with respect to Ξ Ξ can be simpl expressed as Eq. (6-7). P P Ξ P P P Ξ G O G R G G R G P P G O G G R G P O G O G R (6) (7) A e and A p are the design matrix derived from the partial differentiation of the collinearit equations corresponding to the tie points with respect to the parameters and ; z is the observation vector of the EOP provided b the GPS/INS sstem; z is the observation vector of the GP provided b the total station and GPS; K and K are the design matrix associated the constraints; and e z are the error vectors associated with the corresponding observation vectors; σ is the unknown variance component; P is the cofactor matrix of which is generall expressed as an identit matrix; finall P z is the cofactor matrix of reflecting the precision of the GPS/INS data and P z is the cofactor matrix of e z reflecting the precision of the GPs. The mathematical model in Eq. () is the most complete one called Tpe D in this paper. With the different selections of the subsets of the unknowns and constraints we can present other three models Tpe A-. In Tpe A we estimate onl the GPs with the fixed EOPs derived from GPS/INS data without stochastic constraints. In Tpe B to D we estimate both GPs and EOPs. As stochastic constraints we use onl GPS/INS data in Tpe B and onl GPs in Tpe. Both kinds of constraints are used in Tpe D. The main characteristics and mathematical models are summarized in Table and respectivel. Unknowns to be estimated onstraints to be used GP EOP GPS/INS GP A O X X X B O O O X O O X O D O O O O Table. haracteristics of different mathematical models The differentiation of P with respect to Ξ 3 is expressed as Eq. (8-9) where R x R and R z are the rotation matrix for each axis. P G R Ξ 3 G P Ξ 3 G O G R ω G R ϕ G R κ G P G O (8) G R R x (ω)r (ϕ)r z (κ) (9) A B D z A e A p K z z Mathematical Models A p + ~ σ P z A e A p K A e A p K K e z e z e z e z ~ σ ~ σ P P z ~ σ P P z P P z P z.3 Mathematical Models with Four Tpes The entire linearized mathematical models for BBA is expressed as z z A e A p K K + e z e z ~ σ P P Z P Z () where and are the parameter vectors for the EOP and GP respectivel; is the observation vector for the tie points; Table. Mathematical models of different tpes 3. Data Acquisition 3. EXPERIMENATAL RESULTS We acquired the experimental data using a GMMS equipped with an omni-directional camera and a GPS/INS sstem. This camera is actuall integrated with six single frame cameras. Its field of view is 36 and 8 in horizontal and vertical direction respectivel. The size of an original image is 6(H) b (V) pixels while the size of an output integrated image is 54(H) b 7(V) pixels. Each pixel covers /5 deg in each direction. The specification of

4 Ladbug our selected omni-directional camera model is shown in Figure (Point Gre Research 8). The example of omni-directional image captured b the GMMS is shown as Figure 3. We also obtained the position and attitude of the camera when driving along a street using a GPS/INS sstem. The specification of GPS/INS sstem is summarized as Table 3 (Appanix 9). Figure 4. Map of the test area ID 3 45 Figure. Ladbug and GMMS GPs ID GPs Figure 3. Example of omni-directional images XY Position Error (m). Z Position Error (m).5 Roll and Pitch Error ( ). True Heading Error ( ).5 Figure 5. Locations of ground control/check points Image 9 th Image 3 rd Image Tie Points Table 3. Specification of GPS/INS sstems (POS-LV) 3. Data Preparation In this stud we used 4 successive omni-directional images of an area near a road in front of a cit hall in a small cit Osan in Korea. The whole distance between positions of first image and last image is about m. The data were obtained b GMMS from east to west side as depicted b thellow arrow in Figure 4. 3 GPs were measured b a static GPS and total station. Some of them were used as ground control points and the others for ground check points. Figure 5 shows the locations and indexes of ground control points (red) and check points (black). In addition sufficient number of tie points is manuall selected to link the successive images. An example of a pair of tie points is presented in Figure 6. Figure 6. A pair of tie points The total number of GPs corresponding to all the tie points is 8. Each GP appears in at least 4 images up to images. The average number of GP overlapped on two successive images is approximatel 7.9. The details are summarized in Table 4. The existence of each GP along the images is summarized in Figure 7. The row indexes represent the GP indexes where thellow marked indexes indicate the GPs with known coordinates to be used for ground control/check points. The column indexes are the image indexes. Red color indicates the existence of a GP in the corresponding images. For example the st GP exists in the st image not in the nd image.

5 3.3 Result and Analsis Experiments have been conducted with real data for the evaluation of the proposed BBA method with the mathematical models of four different tpes. The GPs estimated from BBA were compared with those measured b a statics GPS and a total station. The GP errors from this comparison are described in Table 5. Figure 7. Existence of GPs in images The RMSE from Tpe B using onl EOP constraints is not improved even in comparison with that from Tpe A using no constraint with fixed EOPs. This means that the EOPs estimated in Tpe B are almost the same as the EOPs initiall provided from GPS/INS data used as fixed EOPs in Tpe A. This must be caused from the assumption of the significantl larger errors in image point measurement comparing to the GPS/INS errors. Hence the relative orientation determined from TPs cannot improve the EOP provided from GPS/INS data. Parameters Value Distance between Images (m) 4 No. Images 4 No. ground control points 4 No. ground check points 9 No. ground points 8 No. image points Avg. no. image per GP 7.6 Avg. no. GP per image 8.8 Avg. no. TP pairs per two successive images 7.9 Table 4. Summar of the experimental data Finall we determined the variance-covariance parameters for the mathematical model of BBA. We assumed that the image point measurement error of deg the GP and GPS errors of 5 cm and the INS error of.5 deg. ID Tpe A : No onstraints and Fixed EOP Tpe B : Onl EOP onstraints In Tpe we performed the BBA with onl 4 GPs (ID 4 8 and ) without an EOP constraints. The RMSE of tpe is significantl reduced comparing with the errors from other tpes. We can determine GPs with a RMSE of less than 5 cm. According to this result GPs can be estimated ver accuratel with a small number of GPs. To use Tpe however some GPs have to be measured accuratel in advance. For Tpe D we use 4 GPs and EOP constraints. The RMSE is reduced slightl comparing with Tpe A and B but not large enough when comparing with Tpe. According to the test results we can infer that EOPs contain sstematic bias and it could be propagated from the difference between the GPS/INS coordinate sstem and S. In future the conformal transformation parameters between the two coordinate sstems will be estimated simultaneousl during the BBA process. The residuals between the measured image points (IPs) and adjusted IPs are shown in Figure 8 for all four tpes. Large residuals indicate the large difference between the measured Tpe : Onl GP onstraints Tpe D : Both EOP and GP onstraints ΔX ΔY ΔX ΔY ΔZ ΔZ ΔY ΔZ ΔZ ΔX ΔY ΔZ AVG STD RMSE Table 5. omparison of GP errors for Tpe A-D (unit: m)

6 z and adjusted IPs and EOPs or GPs ma not be estimated accuratel. According to the statistics of residual for 4 tpes shown in Table 6 the RMS value of the residuals from Tpe are remarkabl small in comparison with those from other tpes. This indicates that the mathematical model of Tpe is the most accurate. Estimated GPs EOPs and GPs for Tpe D are shown in Figure 9 where the blue points are the estimated GPs; the red points are the estimated EOPs; the red circle are ground control points; the green circle is the ground check points Tpe A Tpe Tpe B Tpe D Figure 8. Residuals between measured and adjusted IPs Tpe Min Max AVG STD RMS A B D Table 6. Statistics on residuals x Figure 9. Plot of EOPs (projection centers) GPs and GPs Estimated GP Estimated EOP Ground ontrol Point Ground heck Point ONLUSIONS In this paper we developed a BBA method for omnidirectional images obtained from a ground mobile mapping sstem. We derived new collinearit equations for omnidirectional images and proposed the mathematical models with four tpes for BBA. The proposed method has been tested with real urban data and could successfull estimate GPs using a small number of GPs with a RMSE of less than 5cm. This stud shows that we can accuratel determine EOPs and GPs from omni-directional images with the accurac level required for precision road-side 3D mapping. Furthermore this stud will be efficientl utilized to generate more realistic street view and 3D urban modelling. In future we plan to perform more sophisticated lens calibration for the omni-directional camera to improve the georeferencing accurac of the omni-directional images. AKNOWLEDGEMENTS This research was supported b a grant (7KLSG3) from utting-edge Urban Development Korean Land Spatialization Research Project funded b the Ministr of Land Transport and Maritime Affairs. REFERENES Aizawa K. 4. Image processing technologies: algorithms sensors and applications Signal Processing and ommunications Series New York pp Applanix 9. POSLV specifications Applanix Inc. anada (accessed 4 Mar. ) Beauchemin S. S.. Modeling and removing radial and tangential distortions in spherical lenses In: Multi-Image Analsis Lecture Notes in omputer Science vol. 3 pp.. hoi K. 9. Image georeferencing using at without GPs for a UAV-based low-cost multisensor sstem. Korean Societ of Surveing Geodes phtogrammetr and artograph 7()pp Gontran H. 3. A mobile mapping sstem for road data capture via a single camera In: Proceedings of the Optical 3D onference Zurich Switzerland. Point Gre Research 8. Ladbug specifications Point Gre Research Inc. anada (accessed 6 Apr. ) Silpa-Anan.. 5. Visual localization and loopback detection with a high resolution omnidirectional camera Proceedings of the Omnivis workshop Workshop on Omnidirectional Vision. Tao V.. 5. Mobile mapping technolog for road network data acquisition Proc. Journal of Geospatial Engineering () pp.-3.