SYSTEMATIC CALIBRATION FOR A BACKPACKED SPHERICAL PHOTOGRAMMETRY IMAGING SYSTEM

Transcription

1 SYSTEMATIC CALIBRATION FOR A BACKPACKED SPHERICAL PHOTOGRAMMETRY IMAGING SYSTEM J. Y. Rau a, *, B. W. Su b, K. W. Hsiao c and J. P. Jhan d Departent of Geoatics, National Cheng Kung University, Tainan, Taiwan a, * jyrau@ail.ncku.edu.tw, b steven01357@gail.co, c hq9862@gail.co, d riddle0104@hotail.co Coission ICWG I/Va KEY WORDS: Mobile Mapping Syste, Calibration, Spherical Photograetry ABSTRACT: A spherical caera can observe the environent for alost 720 degrees field of view in one shoot, which is useful for augented reality, environent docuentation, or obile apping applications. This paper ais to develop a spherical photograetry iaging syste for the purpose of 3D easureent through a backpacked obile apping syste (MMS). The used equipent contains a Ladybug-5 spherical caera, a tactical grade positioning and orientation syste (POS), i.e. SPAN-CPT, and an odoeter, etc. This research ais to directly apply photograetric space intersection technique for 3D apping fro a spherical iage stereo-pair. For this purpose, several systeatic calibration procedures are required, including lens distortion calibration, relative orientation calibration, boresight calibration for direct georeferencing, and spherical iage calibration. The lens distortion is serious on the ladybug-5 caera s original 6 iages. Meanwhile, for spherical iage osaicking fro these original 6 iages, we propose the use of their relative orientation and correct their lens distortion at the sae tie. However, the constructed spherical iage still contains systeatic error, which will reduce the 3D easureent accuracy. Later for direct georeferencing purpose, we need to establish a ground control field for boresight/lever-ar calibration. Then, we can apply the calibrated paraeters to obtain the exterior orientation paraeters (EOPs) of all spherical iages. In the end, the 3D positioning accuracy after space intersection will be evaluated, including EOPs obtained by structure fro otion ethod. 1. INTRODUCTION In contrast to the traditional stereo pairs of iages, the spherical iages can provide ore coprehensive geospatial inforation. Exaples can be found such as Google Map street view or indoor panoraic iage for real estate transaction purposes. The spherical iage registration algorith is the ost iportant part since it will influence not only the quality of registration but also the systeatic error. Traditional spherical iage registrations are ainly divided into two categories. The first one utilize iage atching in the overlapped area to estiate iage transforation coefficients. Many algoriths on feature point atching are investigated. Wang et al. (2013) odified the SIFT (Lowe, 2004) algorith to extract conjugate feature points for panoraic iage stitching. However, if the iages contain hoogeneous area such as water or sky, this kind of ethod ight not work well due to too few or not well distributed feature points. The second category define a radius of a virtual sphere and projecting the original frae iage to the virtual sphere. Liu et al. (2012) projected the iages onto a sphere with a radius equal to the focal length of the caera. The result will be affected by a pre-defined sphere radius and the distance between the observation viewpoint and spherical iage. The software provides by Point Grey Researches also use a sphere with a predefined radius, then it projects each iage onto the spherical odel. However, on account of the differences between the sphere centre and each perspective centre, different sphere radii will lead to different registration results. Furtherore, different distances of objects will also lead to isregistration in the overlapped area of two neighbour iages. For the purpose of photograetric easureent, this paper focus on the issues of spherical iage registration ethod to solve the probles entioned above. At first, the spherical caera is calibrated to obtain the interior/exterior orientation of each caera by using an indoor calibration field. Then, derive the relative orientation paraeters (ROPs) between the reference caera and the others. In the end, we project each iage on a virtual sphere with any radius by their relative orientation, interior and exterior orientation. Consequently, the spherical iage generation doesn t require feature point atching and regardless the distances of objects. In the experient, this paper will investigate the systeatic errors for error correction purpose. The error vectors can be used in the future for correcting spherical iage to eet the requireent of high accuracy in photograetry such as 3D positioning using space intersection. 2. EQUIPMENT 2.1 Back-pack Mobile Mapping Syste Figure 1 illustrates the proposed back-pack MMS. This syste contains a Ladybug 5 spherical caera, a tactical grade POS syste, i.e. SPAN CPT (including an IMU and a GNSS antenna), a iniature PC and a battery set. They are fixed by a steel box with a steel pole. On the top of this pole, a Ladybug5 caera is placed to take iages with iniu occlusion. The GNSS antenna is set at the iddle of the pole lower the Ladybug5 about 0.4. The other equipent such as SPAN CPT IMU, PC, and battery are all covered by the steel box for water-proof purpose. In the interior of the steel box, the PC is on the top, IMU and controller is in the iddle and the battery is at the botto. There * Corresponding author doi: /isprsarchives-xli-b

2 are any function of this controller, one is the switch that can start or shutdown the power for PC, IMU and Ladybug5, respectively. The other function is the trigger control that can control the iage acquisition frequency or traveling distance interval between two iages. The traveling distance is easured by an odoeter, but unfortunately this inforation does not used in POS solution. by 90 degrees to reduce the correlation between and exterior and interior orientation paraeters. Figure 2 depicts the distribution of all coded targets together with iage location. Figure 2. Coded targets distribution and iage locations for caera calibration. Figure 1. Back-pack obile apping syste 2.2 Ladybug-5 Caera The spherical caera used in this study is the Ladybug5 fro Point Grey Researches copany. It coposed of six Sony ICX655 CCD caeras. The original iage size of each caera is pixels, the pixel size is 3.45μ, and the focal length is 4.4. Hence, the FOV is Five caeras (denoted as Ca0 to Ca4) are rotated with 90 degrees and aligned in a circle horizontally, whereas the Ca5 takes pictures toward the zenith to produce a spherical iage with a total FOV of 360 degrees in horizontal direction and 135 degrees in elevation direction, respectively. Since Ladybug5 utilizes fisheye lens with short focal length, each iage contains large range so that the overlapped area between adjacent iages can be used for sealess osaicking. Meanwhile, its lens distortion is very serious and need to be corrected during the spherical iage stitching step. Therefore, an accurate caera calibration is critical for this purpose. 2.3 POS Syste Together with the Ladybug5, we equip a SPAN CPT POS syste. With the assist of GNSS, the directional accuracy of this syste can reach /0.015 /0.03 in Roll/Pitch/Heading directions, respectively, after post-processing with the GPS differential positioning. 3.1 Caera calibration 3. METHODOLOGY In order to obtain the relative orientation paraeters (ROPs) between two caeras, lens distortion and interior orientation paraeters of each caera, a caera calibration field is designed. We set up ore than two hundred Australis coded targets (Fraser, 1998) on a sealed roo. The coded targets can be detected and recognized autoatically for obtaining precise iage coordinates. In iage acquisition, eight different horizontal positions and three different heights are chosen for taking pictures. In which, soe of the the were rotated with 90 degrees to increase the overlapped area between iages and tilted The interior orientation and exterior orientation of each caera are solved by self-calibration bundle adjustent with additional paraeters. The atheatic odel used is the photograetric collinearity equations as shown in equation (1). In which, the ( x, y) are the adopted additional paraeters as depicted in equation (2) (Fraser, 1997). where f = focal length x, y = original iage coordinates x p, y p = coordinates of principle point x, y = iage coordinates which have been calibrated for lens distortion K 1 ~K 5 = radial lens distortion paraeter P 1, P 2 = decentering lens distortion paraeter X 0, Y 0, Z 0 = instantaneous position of iage in local coordinate syste X, Y, Z = coordinates of iage point (x, y) in object space r = distance fro principle point 11 ~ 33 = eleents of rotation atrix In the calculation of interior orientation, due to the high correlation between radial lens distortion paraeter and high correlation between decentering lens distortion paraeter and principle point, the significant test was perfored to ignore the non-significant paraeter which don t significantly iprove the posteriori standard error of iage coordinates easureent. 3.2 Relative Orientation In this study, Ca0 is defined as the aster or reference caera. After solving the exterior orientation, the rotation angles and (1) (2) doi: /isprsarchives-xli-b

3 spatial offsets of other caeras relative to Ca0 can be obtained by equation (3). C R s C C = s M RM R C C r s C = (r M Cs r M C ) (3) where R = rotation atrix between two caeras C s = slave caera C = aster caera M = local coordinate syste r = relative position vector of two caeras 3.3 Spherical Iage Stitching Since Ca0 is defined as the ater caera, its iage coordinate syste is used to define a spherical odel with any predefined radius. The perspective centre of Ca0 is located in the centre of sphere. Then, we define the y-z plane as equatorial plane and the inus x-axis is defined as the North Pole. Aziuth angle φ is the angle between the observation direction and an object point A. The elevation angle θ is the angle between the equatorial plane and an object point A. Figure 3 illustrates the diagra of spherical odel used in this study. According to this figure, the relationship between spherical coordinate and 3D Cartesian coordinate is shown in equation (4). (0,0) Figure 4. Coverage of all six caeras in 2D spherical iage coordinate syste. This paper assue that the size of a spherical iage is pixels, consequently each pixel contain an IFOV of Through the direct transforation ethod, the light ray is projected fro the sphere centre onto the sphere. In the beginning, the iage of Ca0 is projected onto the sphere using the EOPs of Ca0 with FOV range fro longitude of -36 to +36 and latitude of -45 to 45 with an interval of Further, we use the EOPs of Ca1 derived by the relative orientation to project the iage of Ca1 onto the sphere fro longitude of 36 to +108 and latitude of -45 to 45, and so on. Notice that the derived EOPs for Ca1-Ca5 does not consider their spatial offset with Ca0, because we treat their projection centre sae as Ca0. Once the elevation (θ) and the aziuth (φ) are known, the 3D Cartesian coordinates can be obtained by equation (4) by assuing any radius (r). Further, by eans of collinearity equations, i.e. equation (5), the corresponding caera s iage coordinates can be obtained. As the derived iage coordinates are usually decial, we can thus interpolate the RGB values fro the original iage and fill it into the correspondent position on the 2D spherical iage coordinate syste. In which, the relationship between the 3D and 2D spherical iage coordinates syste are illustrate in Figure 5. The principle point of Ca0 is located at the centre of 2D spherical iage centre with zero aziuth and elevation angles. In other words, the EOPs of Ca0 can be regarded as the EOPs of the generated spherical iage. Figure 3. The suggested spherical odel. X A = r sinθ Y A = r cosθ sinφ (4) Z A = r cosθ cosφ where X A, Y A, Z A = 3D Cartesian coordinates of an object point on the sphere odel r = any predefined sphere radius φ = aziuth angle θ = elevation angle During the spherical iage stitching, we adopt a range of for direct osaicking those five horizontal caeras without overlap and the Ca5 will cover Figure 4 displays the coverage of all six caeras at the 2D spherical iage coordinate syste. where f = focal length x, y = derived iage coordinates X A, Y A, Z A = coordinates of object point X C, Y c, Z c = coordinates of perspective centre 11 ~ 33 = eleents of rotation atrix (5) Figure 5. The relationship between 3D and 2D spherical iage coordinate systes. doi: /isprsarchives-xli-b

4 3.4 DG (Direct Georeferencing) calibration The goal of DG calibration is to solve the lever-ar and boresight angles between IMU and caera. Figure 6 is the scheatic diagra about direct georeferencing. In which, r(t) GPS is the phase centre of GPS antenna at tie t w.r.t the apping frae. R(t) b is the rotation atrix between body frae and apping b frae at tie t. a GPS is the offset between GPS receiver and IMU easured by ground survey. r(t) b is the IMU s position vector in apping frae at tie t, obtained by cobining r(t) GPS and b a GPS. r c is the caera position vector in apping frae calculated by photo-triangulation. Therefore, using r(t) b and r c we can figure out the a b C (Lever-ar) and R b c (Boresight angles). Figure 7. The distribution of spherical iages and control points Outdoor: For positioning accuracy analysis after direct georeferencing, an outdoor study site is created in Guiren capus of NCKU. There are five buildings with 2-6 floors. In order to allow the spherical iages to cover as any targets as possible, we utilize total station to easure lots of natural features on these five buildings wall as control point. The absolute accuracy of control points is 1 c. Figure 8 deonstrates one building within the outdoor study site, in which the red circles are the location of control points. Figure 6. The scheatic diagra of direct georeferencing 3.5 Direct Georeferencing (DG) For the purpose of eergency response, a fast way to obtain EOPs for each iagery is the direct georeferencing (Ip et al., 2004). Through the DG calibration, we obtain the boresight angles and lever-ar between the IMU body-frae and Ca0. After post-processing of POS data, we estiate the track of IMU body-frae. Thus, we can interpolate each trigger event to obtain the body-frae s position and attitude at the caera exposure tie. The EOP of each photo can thus be calculated by applying the boresight angles and lever-ar to the body-frae s position and attitude. Further, we can conduct the space forward intersection to locate all object s 3D coordinates by easuring conjugate points fro ore than two spherical iages. 3.6 Structure fro Motion and Bundle Adjustent For the purpose to obtain accurate EOPs of each iage, particularly in an indoor environent that GNSS signal was blocked, a structure fro otion (SfM) (Häing and Peters, 2010) approach is proposed instead. It utilizes feature point atching technique to find tie points in different spherical iages and applying bundle adjust adjustent to build the relationship between each iages and object space. It eans we can obtain the EOPs and perfor space intersection to easure object s 3D coordinates for positioning error analysis. 4. CASE STUDY 4.1 Study Site Indoor: An indoor study site located within our departent s building is used for spherical iage positioning accuracy assessent. We evenly place several targets, i.e. black square with a white circle in the centre, on the wall as the control points. Figure 7 is the distribution of spherical iages used in this study, they are acquired by traveling between two floors. In which, the blue balls are spherical iages, the blue flags are control points, and the other sall dots are tie-points on the object space. Figure 8. Outdoor study site 4.2 Visual Analysis of Spherical Iagery Figure 9 displays the spherical iage produced by the proposed ethod. In which, two red rectangles are enlarged and shown in Figure 10 and Figure 11, respectively. Fro the result, we can find out that Figure 10 has very well alignent between two neighbour caeras, but not for Figure 11. It shows soe isalignent still exist. However, iage of Ca5 that pointing upward contains ostly sky and only a few feature points can be found within the overlapped area, the proposed ethod still can work well. This ethod can not only overcoe the proble found in the traditional ethod using feature points for transforation but also present a good iage stitching. In the eantie, we try different radii of the virtual sphere during spherical iage stitching, the generated spherical iages are all the sae. It proves that the proposed algorith is independent to object distance. Please notice that in session 3, we utilize the ROPs to estiate the other caeras orientation, but without using the spatial offset. If we apply the spatial offset, the generated spherical iages will be different when using different sphere radii. doi: /isprsarchives-xli-b

5 Figure 9. Spherical iage of the outdoor calibration field Figure 12. Error vectors fro five spherical iages produced by the PGR software. Figure 10. Good alignent between two neighbour caeras. Figure 11. Inaccurate alignent between two neighbour caeras. 4.3 Systeatic Error Analysis Outdoor Calibration Site: In view of the differences between the sphere s centre and the perspective centres of all six caeras, there will be soe systeatic errors in the generated spherical iage. We choose a test field surrounded by buildings containing any ground control points with known 3D ground coordinates and acquire several Labybug-5 iages. Applying all original iages (except Ca5) in a rigorous photo-triangulation process, we obtain accurate EOPs of Ca0-4. In which, Ca0 s EOPs are treated as sae as the spherical iage. Further, all control points can be back-projected onto the spherical iage coordinates syste (i.e. treated as true position) and copared with their corresponding iage coordinates by anual easureent (exist systeatic error) for error analysis. This discrepancy is denoted as error vector, but in 2D spherical iage coordinate syste. dx R 11 R 12 R 13 X A X c [ dy] = λ [ R 21 R 22 R 23 ] [ Y A Y c ] (6) dz R 31 R 32 R 33 Z A Z c θ = sin 1 ( dz r ) φ = tan2 1 ( dy dx ) Figure 13. Error vectors fro two spherical iages produced by the proposed ethod Indoor Calibration Site: Since the outdoor calibration site does not cover the whole FOV of all six caeras, especially for Ca5 that is pointing to the sky only has a few control points for error analysis. Meanwhile, the distribution of all controls points are not even. Therefore, we utilize the indoor caera calibration site for further systeatic error analysis. Here the coded targets 3D object coordinates are estiated by bundle adjustent during the caera calibration. Meanwhile, the error analysis is conducted on the original iage coordinates syste, not 2D spherical one. Therefore, we can analyse each caera s systeatic error individually. Figures deonstrate six error vector plots of Ca0-Ca5 acquire at the sae exposure tie, respectively. shows the statistics of all caeras error vector length. It shows that Ca0 has an overall error length with a standard deviation of 0.4 pixel, which is very sall. However, visualize Figure 14, we still can find soe systeatic trend. Figure 11 displays the error vectors fro five spherical iages produced by PGR software using a predefined sphere radius of 10 eters. In which, the error vectors have no systeatic trend and all iages (different colours) are not consistent as well. Figure 13 deonstrate the error vectors fro two spherical iages produced by the proposed ethod. As shown in the figure, the systeatic error can be found w.r.t. each original iage s coverage. The directions and agnitudes are consistent. The ajor reasons that cause these systeatic errors are ainly due to inaccurate lens distortion correction and the perspective centres of all six caeras are not the sae. Figure 14. Error Vector of Ca0 doi: /isprsarchives-xli-b

6 Unit: pixel Min. Max. Mean Std. Dev. Ca Ca Ca Ca Ca Ca Table 1. Statistics of error length for all 6 caeras. Figure 15.Error Vector of Ca1 Observing Figures and Table 1, we can also find out clear systeatic errors exist in each caera and their quantities are all large. The systeatic trends for all caeras are different, eaning that their correction has to be perfored individually. If these trends for all iages acquired fro the sae caera are all the sae, we ay thus utilize different 2D transforation functions to correct the during the generation of spherical iage. 4.4 Positioning Error Analysis Here we copare the positioning accuracy using EOPs derived by SfM and DG ethods, individually. Figure 16.Error Vector of Ca2 Figure 17.Error Vector of Ca3 Figure 18.Error Vector of Ca Indoor (SfM Method): Thirteen targets are paste on the wall as the control points. Table 2 shows the object space and iage coordinate errors of each control point, including the RMS to depict the overall accuracy. Point ID Error in object space () Error in iage space (pixel) CP CP CP CP CP CP CP CP CP CP CP CP CP RMS Table 2. The accuracy after structure fro otion. In Table 2, ost control points errors are less than 0.05 but three of the, i.e. CP8, CP9 and CP10, have larger error about It s a two floors indoor study site, so we collect the data in second floor first and then take the stair down to the first floor. Those three control points are located in the iddle of first and second floors. The reasons that cause larger error is the weak geoetry as well as the stitching procedure to generate the spherical iages. In the eantie, Table 2 also illustrates the positioning error in iage space with an overall error of 2.79 pixels. Since the distance fro wall to the caera varied fro 2-10 eters, its spatial resolution is ranging fro Thus, 2.8 pixels is equivalent to a spatial positioning error around To su up, the overall accuracy is less than which is acceptable in indoor 3D apping application. Figure 19.Error Vector of Ca5 doi: /isprsarchives-xli-b

7 4.4.2 Outdoor (DG Method): For the outdoor experient, we first conduct DG calibration then perfor space intersection positioning accuracy assessent. The purpose of DG calibration is to obtain the boresight angles and lever-ar between IMU and caera. Using the iage atching and bundle adjustent to obtain the exterior orientation of reference caera, i.e. Ca0 that has viewing direction sae as walking. We can copare the exterior orientation of caeras with the IMU body frae and get the lever-ar and boresights angles. Table 3 shows the internal accuracy assessent results after DG calibration. The standard deviation of Phi angles is 1.8. The reason we speculated is the MMS has larger up and down vibration when people carry it and walking around the study site. Lever-ar (c) dx dy dz Mean Std. Dev Boresight angles(degree) do dp dk Mean Std. Dev Table 3. The internal accuracy after DG calibration. After post-processing of GNSS and IMU, the exterior orientation of all photos fro Ca0 can be estiated by applying the calibrated boresight angles and lever-ar to each trigger event. After that, using Ca0 s iage coordinate syste we can define a local 3D spherical coordinate syste. Then, it can be convert to 2D spherical iage coordinates, i.e. horizontal axis is longitude and vertical axis is latitude. It eans that when we easure the spherical iage coordinates, we obtain longitude and latitude coordinates. Then, it produces a ray siilar to frae-based iage. Thus, we can perfor forward space intersection to get 3D coordinates of check points and copare it with its true 3D coordinates. In Table 4, the analysis result illustrates that the largest ean error is approxiately 9 eters in E direction with a high standard deviation of eters. There are soe reasons that result in this error. The first one is the DG calibration error and huan walking otion is not fit with the used Kalan filtering during the POS solution. The second one is the algorith to generate spherical iage that each caera has its own systeatic error. E() N() H() Mean Std. Dev Table 4.The accuracy of 3D forward intersection position 5. APPLICATION This syste has been applied in any situations. For instance, there is an earthquake occurred in Tainan in February 6 th, This earthquake caused 117 peoples die and ore than 30 buildings daaged. The land vehicle MMS is prohibited to enter the disaster zone. The back-pack MMS carried by people can overcoe these drawbacks are taken during the rescue period of a collapsed building, i.e. Weikung Kinglong building. These spherical iages can record the realis in 720 for digital docuentation purpose or post-disaster investigation. Figure 20. The spherical iages taken around the collapsed building. 6. CONCLUSIONS AND FUTURE WORKS In this study, a back-pack obile apping syste is developed and several positioning accuracy assessents are conducted in indoor and outdoor study sites. Spherical iage registration ethod is iportant in that it will influence not only the quality of registration but also the errors of the spherical iage. This paper suggests a spherical odel, a stitching algorith based on relative orientation of all caeras. It is confired that the proposed ethod can overcoe the probles in traditional ethods. It can still work well despite the hoogeneous area and the error doesn t relate to the distance of object point. The error analysis results show that different systeatic errors exits in the original iages. The iage atching and structure fro otion are applied to obtain the exterior orientation of caeras due to the lack of GNSS and the overall positioning accuracy is approxiately 6 c. The outdoor experient utilizes direct georeferencing to get the exterior orientation but the ean error is around 10 which is not feasible in apping applications. In the future, a further investigation of spherical iage stitching algorith that can correct each caera s systeatic errors are required and a better POS solution suitable to back-pack MMS is critical. doi: /isprsarchives-xli-b

8 ACKNOWLEDGEMENTS The authors are grateful for the funding support fro the Ministry of Science and Technology (MOST), Taiwan. REFERENCES Fraser, C.S., Digital caera self-calibration, ISPRS Journal of Photograetry & Reote Sensing, Vol. 52, pp , Fraser, C.S., Autoated Processes in Digital Photograetric Calibration, Orientation, and Triangulation. Digital Signal Processing, 8, Häing, K. and Peters, G., "The structure-fro-otion reconstruction pipeline a survey with focus on short iage sequences". Kybernetika. Ip, A.W.L., El-Sheiy, N., and Mostafa, M.M.R., Syste Perforance Analysis of IMU/DGPS Integrated Syste for Mobile Mapping Syste (MMS), The 4th International Syposiu on Mobile Mapping Technology (MMT 2004), Kuning, China. Liu, S.A., Zhao, L.L., Li, J.S., Yao, P., Wei, T.I., Cai, Q., Multi-resolution Panoraic Modeling Based on Spherical Projective Geoetry, nd International Conference on Coputer Science and Network Technology (ICCSNT), pp Lowe, D. G., Distinctive iage features fro scaleinvariant keypoints, International Journal of Coputer Vision, Vol.60, No.2, pp Micusik, Branislav, and Toas Pajdla "Structure fro otion with wide circular field of view caeras." IEEE Transactions on Pattern Analysis and Machine Intelligence, 28.7: Wang, X. H., Wu, K., Wang, S.Z., Research on Panoraic Iage Registration Approach based on Spherical Model, International Journal of Signal Processing, Iage Processing and Pattern Recognition, Vol.6, No.6, pp doi: /isprsarchives-xli-b