Keywords: smartphone, 3D model, calibration, GNSS, bundle adjustment

Transcription

1 Photogrammetry and Remote Sensing CAPABILITIES OF A SMARTPHONE FOR GEOREFERENCED 3D MODEL CREATION: AN EVALUATION Krzysztof Bakuła 1 Adrian Flasiński 1 1 Warsaw University of Technology, Faculty of Geodesy and Cartography, Department of Photogrammetry, Remote Sensing and Spatial Information Systems; Poland ABSTRACT This paper focuses on assessing capabilities of a smartphone for performing a specific close-range photogrammetry task: creating a georeferenced 3D model. One of the subjects discussed in this paper was the application of non-metric camera and location sensors usefulness in photogrammetric data processing. The calibration of the camera was carried out on the basis of measurements in a three-dimensional test field. Another piece of research was carried out to evaluate the quality of angular sensors and GNSS antenna which are embedded in the tested device. Data collected in these experiments was used for image geometric correction and weighting the equations in the adjustment process. In a further experiment, an evaluation of smartphones capabilities for creating a georeferenced 3D model was made by a creating model of an Old Town wall fragment located in Warsaw. During several bundle adjustments with different options it was proved that initial values of exterior orientation can increase the accuracy of photogrammetric post processing with a low number of control points. Application of integrated sensors in smarthphones is definitely useful for low-cost 3D modelling and it can be a good solution for achieving accuracy of decimeters in creating a model of an object. However, accuracy of centimeters required in close-range photogrammetry cannot be guaranteed by using onlya smartphone without considering an application of at least 3 control points. Keywords: smartphone, 3D model, calibration, GNSS, bundle adjustment 1. INTRODUCTION The capabilities of modern mobile phones are constantly increasing which have led to the creation of a new group of these devices: smartphones. A multitude of their functions and their ever-increasing popularity and availability raises the question of whether it is possible to use a smartphone as a low-cost, mobile tool in photogrammetry. Smartphones owe their nickname mainly due to the combination of multiple functions in one device. Smartphones are usually equipped with a digital camera, a GNSS receiver, compass, accelerometers, Internet connectivity and other components that are applicable to both business users as well as to those whose smartphone is mainly used just for entertainment. Despite amateur-purpose of digital camera and GNSS receiver, which smartphones are equipped with, many research groups have conducted an evaluation of their suitability far beyond the standard use of these devices. As examples of such applications, attempts at creating georeferenced 3D models based on known 85

2 13 th International Multidisciplinary Scientific GeoConference SGEM 2013 photogrammetric algorithms should be mentioned here. Such models from smartphonebased data are usually generated in postprocessing. However, there is also some software that is able to determine the measurements using stereoscopic images to calculate certain distance on the basis of recorded approximated exterior orientation (EO) of sensors. The development of such studies is not only linked to the development of smartphones. Several studies have been already carried out with first phones equipped with a digital camera as a sensor for images recording that provides a low-cost photogrammetric product of lower accuracy [1], [9]. While technology of smartphones was developed, this type of device has also become a photogrammetric tool [4], [8], which is used even in UAV platforms [10], [2]. Such an application is not only limited to optical image acquisition they can also be used for platform control. Smartphones as cheap sensors may replace other technical solutions combining non-metric cameras with GNSS receivers and low-cost attitude and heading reference system (AHRS) - micro-electro-mechanical system (MEMS) that gives information about the orientation of the camera. The example of such a combination is presented in [5]. High hopes are also related to application of smarthophones in low-cost 3D modelling providing data from videos sequential images [6]. The presented paper is an example of application of images and registered exterior orientation for the photogrammetric study of an architectural object. The aim of the study is also to examine the sensors that the analyzed smartphone is equipped with. The intended result of the work was to create a three-dimensional object model and analyze the influence of geo-locational and angular sensors in accuracy of model georefencing. Such an analysis is part of the modern trends of the development of engineering techniques, which are also increasingly available for non-specialists. Their skillful use can provide a lot of information in many geomatics applications. a) b) Fig. 1. LG P920 Swift 3D (a) and applied 3D calibration field (b) 2. EVALUATION OF SENSORS The aim of the first part of the study was the analysis of sensors in smartphones. Determination of sensors accuracy, their stability and the possibility of potential removing of systematic errors are crucial to assess smartphone capabilities such as the elimination of image distortion caused by systematic errors of the lens and later weighting of observation equations during bundle adjustment. All sensors which were used in smartphone software in later works have been subjected to research tests. First of all calibration of a non-metric camera was performed. The following analysis 86

3 Photogrammetry and Remote Sensing concerned the test of accelerometer measurements of pitch and roll, magnetic tests measure of azimuth and finally GNSS receiver testing. In presented research smartphone LG P920 Swift 3D (figure 1a) was used for evaluation of its capabilities in photogrammetric tasks. This device is equipped with a GNSS receiver, gyroscope, accelerometer and magnetometer as well as a digital camera with a resolution of 5 megapixels (2592 x 1944 pixels). The size of matrix element is slightly more than 2 microns. LG P920 Swift 3D debuted on the market in 2011 as a device that has a display capable of creating the effect of image depth without additional equipment (such as special glasses) which is achieved by using parallax barrier technology. It is also capable of capturing stereoscopic images. The short base and consequent small ratio of object distance to base length do not guarantee accuracy in a direction parallel to optical axis and from a theoretical point of view, do not ensure high accuracy, providing 3D model only for visualization especially for objects photographed from a short distance Camera calibration. The calibration of the smartphone was carried out by photogrammetric spatial resection of 3 photos with 3 independent measurements series. During this process the following values were determined: the position of the principal point, focal length, 3 parameters of radial distortion, 2 parameters of tangential distortion and 2 afinity parameters. The essence of such a measurement is based on high redudancy of observations, which during the calculation of exterior orientation parameters also provides parameters of camera calibration using the rigorous formulas. Based on coordinates of points in a three-dimensional test field (figure 1b) and measured fiducial coordinates, the program for camera calibration defined exterior orientation (coordinates of projection center - X C, Y C, Z C, rotation angles of image - ω, φ, κ), interior orientation parameters (position of principal point - x 0, y 0 and focal length C k ) as well as parameters describing radial and tangential distortion (K 1, K 2, K 3, P 1, P 2 ), affinity (A, B) [3]. Avarage calibration parameters are shown in table 1. 1 pixel 10 pixels Fig. 2. Visualization of the impact of radial (a) and tangential (2) distortion for analyzed smarthphone LG P920 Swift 3D 87

4 13 th International Multidisciplinary Scientific GeoConference SGEM 2013 The example of distortion values is shown in figure 2. The maximum image distortion caused by the radial distortion was 13 pixels (30 microns). Another factor deforming images obtained from the camera is affinity, which in the case of the LG P920 is quite large. Its occurrence may be caused by the shape of the light-sensitive sensor in the camera matrix, which is mainly rectangular rather than square [7]. Such a statement cannot be certain because of the lack of detailed data about the structure of the matrix used in the test device. The size and direction of the vectors associated with affinity is shown in figure 3. Tab. 1. Parameters calculated in calibration procedures for analyzed smarthphone LG P920 Swift 3D Fig. 3. Visualization of affinity impact for analyzed smarthphone 2.2. GNSS testing x 0 y o C k [px] [px] [px] A -0,00012 B -0,01602 K 1-1,6334 E-08 K 2 6,6654 E-15 K 3 P 1 P 2-3,0499E-22-4,1830E-07-4,4982E-07 GNSS receiver tests relied on measurements of the absolute error of sensor position. In experiments differential GNSS/RTK observations were used as the reference data. The example test results are shown in figure 4 presenting the accuracy to a few meters. On the basis of all data obtained in this study (triple measurements of 4 points at different epochs), the root-mean square error (RMSE) of sensor position was calculated as 4.30 m. Altitude accuracy was estimated empirically as twice as low Angular sensors testing Fig. 4. The results of absolute accuracy of GNSS sensor in analyzed smartphone Due to the integration of sensors in angular measurements in smartphones (magnetometer, accelerometers, gyroscope), tests were carried out in two stages: separately for ω (pitch), κ (roll), which are measured using a set of accelerometers and a 88

5 Photogrammetry and Remote Sensing gyroscope, and separately for φ (azimuth), which is measured with the influence of magnetics. In analysis considering ω and κ angles, several tests were conducted including both continuous and discontinuous angle changes, with true values of measured angle read from a tilting table providing the value of angular error of sensor in size 1. Contrary to that analysis, error of φ angle was estimated in surveying measurement in comparison with results from total station. Such an observation was carried out far away from metal objects because of sensitivity of the magnetic sensor in the presence of and its erroneous work in closed areas. Error of φ angle was a much higher value than ω and κ error and was estimated to a few degrees. The magnetometer did not show enough accuracy in short time angle changes which leads to the conclusion that its measurements were unstable. 3. PHOTOGRAMMETRIC PROCESSING In an experiment, an evaluation of smartphones capability for creating a georeferenced 3D model has been carried out by creating a model of Old Town wall fragment located in Warsaw. Control points (GCPs) network was set up on the object and approximate exterior orientation parameters were recorded during taking photographs for further georeferencing the model. Overlap for photos was about 80%. Photogrammetric postprocessing was done in Intergraph software. Bundle adjustment was done several times with different options (with or without usage of direct georeferencing) for accuracy analysis of determined exterior orientation parameters. Finally, a photorealistic model was created from point cloud generated from stereoscopic models Bundle adjustment Bundle-adjustment (in this case it can also be named as terra-triangulation) was carried out in several variants depending on the alignment parameters, a number of control points used as well as approximate observations from sensors measuring the location and inclination of the device. In the first reference variant, (named "0") the bundle adjustment was carried out without initial values of EO from smartphone. Images were registered and they created a strip based on regularly distributed 18 tie points. The next step was a determination of exterior orientation using 6 control points located in the object. The remaining 5 points were used as check points. In another approach it was decided to use an image with approximate exterior orientation elements. The file containing coordinates of the projection centers and angular elements of EO including their estimated errors was imported. Planimetric error for position of projection centers was estimated as 4.5 m, vertical 9 m, ω, κ error as 1 and φ error due to low accuracy was not considered as an initial parameter. In variant "1" of the analysis raw data about EO from sensors in the smartphone were included in adjustments. There were no operations other than the measurement of check points to assess calculated error in spatial resection. It was noticed that in such an approach high transverse parallax remained. In order to improve accuracy related to Z coordinate, relative orientation was carried out and these results was considered as 2 variant. Three following variants of bundle adjustment, besides using initial EO parameters and processing relative orientation, were carried out with additional respectively 1, 2 and 3 control points (variants: 3, 4, 5 ). The last variant n was the summary of the research. The same location of ground control and 89

6 13 th International Multidisciplinary Scientific GeoConference SGEM 2013 check points was used (6 control points and 5 check points) in this approach, but in this alignment there were also initial exterior orientation elements from the smartphone. The results of described variants of bundle adjustment are shown in table 2. Variant Number of control points Tab. 2. Results of bundle adjustments in various variants Number of check points RMSE for control points RMSE for check points mx [m] mz [m] my [m] mx [m] mz [m] my [m] n D modelling In the following evaluation of capabilities of the smartphone in photogrammetric issue, the raw point clouds were generated from each stereoscopic model based on image matching. After merging of point clouds and their automatic filtering, a 3D model of the object was created as a digital surface model in TIN structure. The next step was to provide textures for a generated model which was carried out in external software. The result of modelling can be seen on figure 5. It is worth noting that this model is not subject to any correction or participation of manual measurement of breaklines which is observed as problematic edge lines of detail. Such a presentation, however, could be easily edited by the user and it is an example of automation in the whole experiment. Y Z Fig. 5. Model of Warsaw Old Town Wall façade with photorealistic texture 4. DISCUSSION To summarize, in the evaluation of smartphone optical sensors it can be indicated that the distortion is relatively small - the largest of distortion reached 13 pixels, which is X 90

7 Photogrammetry and Remote Sensing only 30 microns, while the distortion for standard non-metric camera lens can even be up to 100 microns at the image corners [7]. These results are similar to [4]. Due to quite large values of affinity that were noticed it was decided to subject each image to geometric correction including parametric model of distortion and its impact, which is a different approach than presented in [1], [8]. GNSS measurements did not give the possibility of their sole use. Accuracy to several meters does not allow the implementation in photogrammetric study. Only the usage of relative orientation improves accuracy, and during the addition of subsequent GCP increasing of bundle adjustment accuracy was provided. Recorded measurements of initial values of ω and κ angle also had a large influence on time calculation time (fewer number of iterations). Their high measurement accuracy was proved in the presented research. One of the weakest elements of the analyzed device was the possibility of magnetic azimuth measurement - φ angle. This problem was also noted in the experiments presented in [4]. The values obtained in the best variants of bundle adjustment have a very small error - less than 1 cm for the XZ and less than 2.5 cm for the Y. These values are significantly better than those obtained during the tests of a few selected mobile phones presented in [1], which is associated with smaller size of matrix in these devices (0.3 and 1.2 megapixels). That also indicates the development of optical sensors in smartphones. The results of low-cost approach based only on GCP are satisfactory, despite the fact that the error in the XZ plane is about 2.5 times larger than the size of pixel - ground sampling distance (GSD) (approximately 4.4 mm). Large errors were noticed in the variants that are based solely on initial values of EO captured by smartphone ("1" and "2"). These variants provided a worse result than reference variant without initial EO, which was calculated by GCPs measurements. This fact was caused by low accuracy GNSS receiver implemented in device especially considering vertical error (Z coordinate). Currently, there are several research programmes underway to enhance the accuracy of observations recorded by smartphones such as including real-time kinematic (RTK) observations (European Geostationary Navigation Overlay Service - EGNOS - providing corrections from satellites for various devices or mrtk providing corrections from ASG - EUPOS network for mobile devices). Application of only on control point (located in the object center) allows for a 4-time increase in the accuracy of image orientation, which provides decimeters georeferening of object model. This is a significant improvement of the results. Such approach maintain also a minimum amount of practical work. The use of three control points results in a centimeter accuracy of bundle adjustment. 5. CONCLUSSION The possibility of using smartphones to create georeferenced 3D models can definitely be evaluated as positive. Constantly increasing resolution of CCD matrix allows more and more detailed images to be captured. Although solely the use of GNSS angular data directly from the device does not provide satisfactory accuracy in the determination of exterior orientation parameters, the application of only three, regularly distributed control points can guarantee the achievement of accuracy to a few centimetres. Including additional data about exterior orientation derived from a smartphone in the calculation enables a significant reduction of required field work 91

8 13 th International Multidisciplinary Scientific GeoConference SGEM 2013 related to a large number of control points. In the presented analysis the usefulness of images acquired by smartphone was not a main subject in this paper, because their potential easily allowed point clouds to be generated and the object model to be created. Considering the history of mobile devices, further and more rapid development and higher significance of low-cost devices such as smartphones and tablets can be expected. Their increasing computational opportunities and growing memory allow for the hypothesis that it will be possible to perform photogrammetric studies, similar to that shown in this paper, using only a smartphone and mobile applications in the near future. The world's research is continuing which confirms the photogrammetric potential of mobile devices in 3D model creation. Further works with different applications [8] are also planned by many scientific institutions. However, depending on the required accuracy, a smartphone can be definitely considered today as a sensor representing a high potential in the modeling of 3D objects for issues related to close range photogrammetry. REFERENCES [1] Agapiou A. & Georgeopoulos A. Photogrammetric Potential of Digital Camera in Handheld Gadgets for Digital Close Range Application. The 7thInternational Symposium od Virtual Reality, Archeology and Cultural Heritage VAST, Cypr [2] Bakuła K., Ostrowski W. The application of non-metric digital camera in aerial photogrammetry on selected examples. Archives of Photogrammetry, Cartography and Remote Sensing, vol. 24, pp , [3] Bujakiewicz A., Kowalczyk M., Podlasiak P. & Zawieska D. Calibration of Very Close Range Digital Cameras. Journal of PAN Geodesy and Cartography vol.55., No. 2 pp , [4] Khosravani A.M, Boehm J & Fritsch D. Potential of Smartphones as Multi-Sensor Systems in Low-Cost Close-Range Photogrammetric Projects. Proceedings SPAR Europe [5] Kolecki J. & Kuras P. Low cost attitude and heading sensors in terrestrial photogrammetry - calibration and testing, Archives of Photogrammetry, Cartography and Remote Sensing, Vol. 22, pp , [6] Kowalczyk M. Evaluation of sequential images for photogrammetrically point determination. Archives of Photogrammetry, Cartography and Remote Sensing, Vol. 22, 2011, pp , [7] Luhmann, T. Close range photogrammetry: principles, techniques and applications. Dunbeath : Whittles Publishing, [8] Smith, M. & Kokkas N. Assessing the photogrammetric potential of cameras in portable devices. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XXXIX-B5, pp , [9] Tokarczyk R., Kolecki J., Tokarczyk P., Application of mobile digital camera in 3D visualization of shrine. Archives of Photogrammetry, Cartography and Remote Sensing, Vol. 17, pp , [10] Yun M.H., Kim J., Seo D., Lee J., Choi C. Application possibility of smartphone as payload for photogrammetric UAV system. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX- B4, Melbourne, Australia,