INVESTIGATION INTO THE USE OF ACTION CAMERAS IN THE DOCUMENTATION OF ARCHITECTURAL DETAILS - THE CASE STUDY OF A BAROQUE CHAMBER.

Transcription

1 INVESTIGATION INTO THE USE OF ACTION CAMERAS IN THE DOCUMENTATION OF ARCHITECTURAL DETAILS - THE CASE STUDY OF A BAROQUE CHAMBER. J. Markiewicz 1*, S. Łapiński 2, M. Pilarska 1, R. Bieńkowski 3, A. Kaliszewska 3 1 Faculty of Geodesy and Cartography, Institute of Photogrammetry, Remote Sensing and Spatial Information Systems, Warsaw University of Technology, Warsaw, Poland - (jakub.markiewicz, magdalena.pilarska)@pw.edu.pl 2 Faculty of Geodesy and Cartography, Division of Engineering Geodesy and Control Surveying System, Warsaw University of Technology, Warsaw, Poland - slawomir.lapinski@pw.edu.pl 3 System Research Institute, Polish Academy of Sciences - (Rafal.Bienkowski, Agnieszka.Kaliszewska)@ibspan.waw.pl Commission II, WG II/8 KEY WORDS: low-cost photogrammetry, Xiaomi camera, accuracy, shape analysis, cultural heritage, frescoes ABSTRACT In this paper the possibility of using the Xiaomi 4K action cameras as a low-cost sensor for the generation of high resolution documentation of architecture and architectural elements in the field of Cultural Heritage was analysed. For that purpose a series of images was acquired together with tachometric measurements to determine the ground control points. Additionally TLS data was collected, which was treated as a reference. For the purpose of point cloud generation the Structure-from-motion (SfM) and Multi- View Stereo (MVS) approaches were used. The following parameters of the collected data and the resulting documentation were tested: the interior orientation parameters analysis, quality of the Xiaomi built-in Lenses Distortion Correction; the accuracy of the orientation on ground control and check points, the point cloud density; the flatness of the walls; the discrepancies between point clouds derived from the low-cost cameras and TLS data, shape of the architectural details based on cross-section analysis. After the analysis of the obtained results it can be concluded that the Xiaomi 4K low-cost sensors are well suited for the purpose of documentation of architecture and architectural details. All the data for the presented investigation were acquired at the baroque residence of the Bieliński Palace in Otwock Wielki in Poland. 1. INTRODUCTION The use of documentation obtained through laser scanning and photogrammetric approach is a well-established practice in all fields dealing with architecture and architectural details such as Art History, Archaeology, or Cultural Heritage Management and Preservation. This type of documentation can be gathered with the use of different sensors, such as a wide array of cameras available to investigators or lasers scanners. Generation of precise and, at the same time, high resolution architectural documentation, acquired on the basis of terrestrial laser scanning or dense point clouds from digital images is a still open issue, which presents many challenges. Although data processing algorithms are intensively developed with consideration of high resolution orthoimage processing or generation of accurate vector drawings, limitations concerning the utilisation of measuring sensors are still the issue which should be solved. The choice of the sensor is usually determined by the needs of the specific project and the funds available. There is a general consensus that the higher the resolution of the sensor the higher the quality of the final documentation. In the presented paper we investigate the possibilities of using low-cost sensors, in our case the Xiaomi 4K action cameras for the documentation of complex architectural details with the application to Cultural Heritage Management and Preservation. In order to gain an insight into the metric properties of the documentation we compare it with the results obtained through the use of a phasebased laser scanner Z+F 5006h (treated as a reference data). In order to validate the possibility of using Xiaomi 4K low-cost cameras in Cultural Heritage documentation, following parameters were analysed: the interior orientation parameters analysis, quality assessment of Xiaomi camera built-in Lenses Distortion Correction, the accuracy of the orientation on ground control and check points, the point cloud density analysis, the flatness of the walls, the discrepancies between point clouds from the low-cost cameras and TLS data, shape of the architectural details based on cross-section analysis. The results presented in this article are the outcome of the investigation by the interdisciplinary team of researchers from the Warsaw University of Technology and the Systems Research Institute of the Polish Academy of Sciences, carried out at the Bieliński Palace in Otwock Wielki in Poland. This investigation is a continuation of previous works on the use of close-range photogrammetry in fresco documentation (Markiewicz, et. al., 2017). There following problems were discussed: the possibility of using the low-cost Xiaomi cameras (2K and 4K) in frescoes documentation, the quality of the point cloud based on flatness analysis, and comparison of point cloud obtained from Xiaomi with the point cloud from full-frame camera. After the analysis authors have chosen the Xiaomi 4K camera for the following experiment because of its higher quality. In this article the test field is characterized by complex shapes such as decorated fireplace with two antithetic figures of winged female sphinxes and tondo in a decorative stucco frame filled with a now damaged fesco. Authors CC BY 4.0 License. 667

2 2. STATE OF THE ART The development in the field of computer vision makes it possible to generate a photogrammetric documentation using an unordered set of digital images. As a tool for Cultural Heritage, photogrammetry is appreciated for being relatively easy and fast, requiring little additional work and tools when compared to traditional methods. The final product in the form of orthoimages allows for additional measurements, not recorded on-site, which is of great importance in the investigation of architecture (Gianniou et al., 2007; Georgopoulos et al., 2004, 2005). The quality of the entry data is of great importance for the quality of the final product, however as it will be shown below, low-cost sensors are a widely available source of images producing reliable results. Additionally, there is a growing number of commercial as well as open-source softwares for processing sets of images and generating orthoimages, making photogrametry a common choice in a wide array of fields (Remondinio el al., 2014, Kersten el al., 2016). The final product and the processed data from these softwares, especially point clouds, can be an important staring point for further analysis (such as angles of structures and surface flatness), which allows for the analysis of the geometrical properties of the documented structures. Low-cost sensors such as action cameras are becoming easily available, however their quality, especially less stable lenses, render a calibration step necessary. In order to address this necessity many cameras are equipped with built in selfcalibration system. By self-calibration is we mean a calculation of the inner orientation parameters, during the adjustment process including the calculation of the object point coordinates and external camera orientation parameters. This method is most effective for digital images (Kraus, 1997; Clarke and Fryer, 1998; Cardenal et al., 2004). The differences in calibration models was widely described by i.e. (Brown, 1971, Zhang and Yao, 2008, Wang, 2012). 3.1 Test field description 3. PERFORMED EXPERIMENT The subject of our investigation is one of the rooms of the Bieliński Palace, near Warsaw (Poland), decorated with frescoes covering all walls of the room and an elaborate Stucco decoration crowning a fireplace. The Palace is located 30 km south from Warsaw, on an artificial island in the oxbow lake of Vistula River. The main part of the palace was built in the 80 s of the XVII century, and later in the 40 s of XVIII wings were added. The Palace is a great example of late baroque noble residence. The investigated room is located on the upper floor of the Palace, in the SE corner of the central part of the palace. Figure 1. The example of decoration used in the process of low-cost Xiaomi 4K camera validation. A) N Wall with frescoes, B) Decorated fireplace with two antithetic figures of winged female sphinxes on the W wall, C) Fresco tondo in a decorative stucco frame above the N door. Figure 2. The distribution of control and check points used in TLS and images orientation. A) Sketch of point distribution, B) The example of natural measured point, C) The example of marked points on dense point cloud. The room is entirely decorated with pale-coloured frescoes running down from the ceiling featuring landscapes with elements of architecture. The room has three doors, one in the centre of N wall, and the and two in the SE corner. The door in the N wall is leading to the central part of the palace and the main staircase, and the doors in the SE corner are leading to a subsequent room and to the tower. Large parts of the E and the S wall are taken up by large windows. The space between the Authors CC BY 4.0 License. 668

3 windows is filled by fresco panels with architectural motives. Larger areas of fresco decoration are located on the W and the N wall. The fireplace is located on the W wall. Its whole decoration is executed in the stucco technique. The fireplace opening is framed by a simple cornice, flanked by spirals. Above the cornice, on a simple ledge rest two antithetic figures of winged female sphinxes (Fig. 1B). On their backs rests a painted fresco tondo in a decorative stucco frame, flanked in the upper part by putti. In our investigations we focus on the figures of the sphinxes (Fig. 1C). The precise details on their wings, as well as the hair and the laurel wreaths, allow us to test the accuracy and the details of the point clouds generated. 3.2 The description of used equipment In the conducted experiment a low-cost Xiaomi 4K camera, Z+F 5006h terrestrial laser scanner were used as well as a Total Station for measuring ground points. The Xiaomi Yi 4K camera has a low-cost sensor with a pixels resolution, the focal-length of 2.68 mm and the pixel size of 1.55 µm. It has a wide angle ( fish-eye ) lens, which influences the radial distortion on the edges of the images. For the purpose of our experiment the camera was mounted on a horizontal bar in order to get a stereoscopic type camera, additionally external light sources were added. Figure 3. The sketch of A) Image distribution, B) TLS positions 4. RESULTS 4.1 Pre-calibration and self-calibration The Xiaomi Yi 4K camera calibration was conducted using the Agisoft Lens, which is an automatic lens calibration software. In this camera calibration approach the computer screen was used as a calibration target. The Xiaomi Yi 4K camera is equipped with Lens Distortion Correction (LDC), Therefore, in the first step 15 distorted images of the calibration target were acquired. For radius equal 1500 pixels, the distortion reached 90 pixels (Fig. 4A). Further 19 undistorted images, with the usage of integrated LDC were taken. For radius equal 1500 pixels, the distortion was between 4 and 5 pixels (Fig. 4B). Point clouds in the investigated interior were obtained using a Z+F 5006h terrestrial laser scanner. The resolution of the acquired data was 6 mm/10m. The maximum angle range of the point clouds was 310/360 degrees. The distance between the scanner position and the object was between 1.5 to 8 m. In order to determine the ground control points the Leica TCRP 1202 total station was used, which allows to measure angle with 2 accuracy and reflectorless distance measurement with 2 mm +/-2 ppm accuracy, respectively. As a result of the observation adjustment the accuracy of 1.5 mm for control and check points was obtained. 3.3 Photogrammetric measurement The measurements begun with the establishment of the 3D control network. The control network was the base for the orientation of the images and comparison of the TLS data with image products (dense point clouds). On each wall few points were measured (at least 6 points on every wall) (Fig. 2A-C). The points should be easily identifiable and distinguishable both on the images and the TLS scan. Additionally, the natural control points should be evenly distributed on every wall. The control points were measured with the use of tacheometry with non-prism distance measurement (Fig. 2B). Simultaneously X, Y and Z coordinates of the measured points were determined. The established control network determines the local coordinate system for the images and TLS data. Figure 4. The sketches of the distortions: A) before B) after build-up Lense Distortion Corrections (LDC). Vertical red line - the half of the sensor size. The results of the calibration shows that the built-in algorithm for image distortion correction, reduces the influence of "fisheye effect" correctly (Fig. 4). Additionally, the parameters of distortion distribution occurring during the self-calibration process were analysed (Fig.5). After the tacheometric measurements of the control and check points were done, 404 images of the room were taken with Xiaomi Yi 4k camera (Fig. 3A). Further, measurements with the terrestrial laser scanner were conducted. All scans were obtained from similar position (Fig. 3B). Authors CC BY 4.0 License. 669

4 Figure 6. Map of point density distribution (10 dm2) for 4 analysed walls. 4.4 Plane flatness analysis Figure 5. The example of distortion distribution after selfcalibration process on undistorted images The analysis of Figure 4 proves that the process of selfcalibration was done correctly. The distortion effect is the biggest on the edges of the images and its distribution is symmetrical, which proves the correctness of the parameters obtained. It should be noticed that this means that this kind of low-cost devices are appropriate for the architecture documentation. 4.2 Image orientation analysis In order to check the quality of image orientation the SfM (Structure from Motion) approach implemented in Agisoft PhotoScan has been used. As reference data, the ground control (20) and check (10) points from the geodetic measurements were used: the RMSE reprojection error was about 2.3 pix, the RMSE on control points for X value was 3.4 mm, Y value 3.5 mm and Z value 2.1 mm, the RMSE on check points for X value was 5.8 mm, Y value 2.9 mm and Z value 3.3 mm, the RMSE related to pixels for control was 0.6 and 1.0. The obtained results shows that quality of the images orientation is very high, so that those cameras might be used in the process of monuments documentation. In the course of the flatness analysis, for chosen architectural elements planes were fitted and differences between the point clouds and the fitted planes were calculated. For that purpose an application in MatLab based on function pcfitplane has been created which allows to automatic fitting of a plane. The main parameters of this function are planemodel object specified as a 1-by-4 vector and maxdistance which determined the distance from an inlier point to the plane. The four parameters [a, b, c, d] describe the equation for a plane: ax+by+cz+d=0 are computed. This function uses the M-estimator SAmple Consensus (MSAC) algorithm to fiting the plane. Figure 7 shows the accuracy of the plane fitting into the SfM point cloud generated from the images taken with the low-cost cameras. The distribution of errors, based on map deviations from the plane, was analysed. It might be noticed that the deviation for Xiaomi 4K MVS point cloud is lower than +/- 12 mm (Fig. 6) and the histogram of the acquired values is similar to the Gaussian (normal) distribution. For all of the cameras deviations are similar and about 95% of the points do not exceed deviation of m. The examined walls have some geometric irregularities because of their historical character. Additionally, on the top of the walls a large deviation might be noticed. This is due to the fact that these parts were covered by a lower number of images and are characterized by worse texture than the rest of the wall. In Figure 7A and D the areas with the incised straight vertical lines were shown, which are the sketches used for frescoes painting. Some deviation might also occur because of the degree of damage of the walls and their partial renovation. 4.3 Point density analysis One of the most important factors, affecting the quality of point clouds is their density. After conducting the orientation of the acquired images, dense point cloud were generated. Two settings of point cloud density were chosen in Agisoft software: High and Ultra High. For each wall of the room densities of the point clouds were analysed, as well as the flatness of the relief stucco decoration and frescos were examined. The mean density of the dense point cloud was approximately 30 points per 10 cm x 10 cm (Fig. 6) cell, but in some places, where the fireplace and bas-relief occur, the point density was higher. Authors CC BY 4.0 License. 670

5 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2, 2018 Figure 7. Map of MVS point deviation from a fitted plane for 4 analysed walls Figure 8. Deviations between point cloud form TLS and Xiaomi 4K camera for figures of winged female sphinxes) X value, B) Y value, C) Z value and D) linear value. 4.5 Deviations between Xiaomi 4K and TLS point clouds The geometric quality of the surfaces obtained from laser scanning was compared with the quality of clouds of points generated from digital images. One of the most important factor which influences the quality of final documentation, such as orthoimages or 3D models, is the accuracy of point cloud. In order to check the accuracy of projection of the sculptures, the differences between the point clouds from Xiaomi 4K and TLS data were analysed. Figures 8-9 shows the results of the comparison. Figures 8 show that the distribution of differences between the two point clouds are similar for all of the directions (X, Y and Z). The most significant deviations are on curved parts and did not exceed +- 8 mm (Fig. 8D). However, the highest deviations occur for X (Fig. 8A) axis and the lowest on for Z direction (Fig. 8C). In the case of the sphinxes figures the mean deviations for X was equal to 0.0 mm, Y mm and Z mm. In table 2 was shows the percentage number of points with specified deviations Table 1. Percentage number of points for specified deviations sphinxes figures. Deviations [mm] -12 : -9-9 : -6-6 : -3-3 : 0 0:3 3:6 6:9 9 : 12 X [%] Y [%] Z [%] L [%] Authors CC BY 4.0 License. 671

6 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2, 2018 To summarize, the Xiaomi point cloud allows to generate a 3D model of complex sculptures with accuracy higher than 6 mm. Similar analysis was performed for fireplace frame. Figure 9 show that the distribution of differences between the two point clouds are similar for all of the directions (X, Y and Z). As well as in the previous test field, the highest deviations occur on curved parts and on those without good, distinguishable pattern and does not exceed mm (Fig. 9D). The highest deviations occur for Y axis (Fig. 9B) and the lowest on for Z direction (Fig. 9C). In the case of the decorated fireplace the mean deviations for X value was equal to 0.1 mm, Y mm and Z 0.0 mm. The result of a detailed analyse was shown in Table 2. To summarize, the Xiaomi point cloud allows to generate a 3D model of fireplace with accuracy higher than 12 mm Table 2. Percentage number of points for specified deviations fireplace Deviations [mm] -12 : -9-9 : -6-6 : -3-3 : 0 0:3 3:6 6:9 9 : 12 X [%] Y [%] Z [%] L [%] Figure 9. Deviations between point cloud form TLS and Xiaomi 4K camera for fireplace: A) X value, B) Y value, C) Z value and D) linear vale and decorative stucco frame: E) X value, F) Y value, G) Z value and H) linear vale. Scale bars for X, Y, Z values (I) and linear deviations (J) The analysis process also covered point clouds acquired for less pronoun relief stucco decorations of complicated geometric features (Fig. 9 E-F). Results of analyses are presented in Fig. 9 E-F; the mean deviations for X value were equal to 2.3 mm, Y 1.1 mm, Z mm. In table 3 the percentage number of points with specified deviations was shown. To summarize, the Xiaomi point cloud allows to generate 3D model of sculpture with accuracy higher than 6 mm. Table 3. Percentage number of points for specified deviations test filed 3 Deviations [mm] -12 : -9-9 : -6-6 : -3-3 : 0 0:3 3:6 6:9 9 : 12 X [%] Y [%] Z [%] L [%] Shape analysis - cross-sections In order to perform an independent analysis of the shape reconstruction, the cross-sections for previously mentioned test field were verified. On the Figure 10 an example of vector drawing was show. The cross-sections based on TLS data was marked as a blue polyline and the cross-sections from Xiaomi point cloud as a red one. In the case of more complex shapes it might be noticed that a Xiaomi vector drawing is generalized and some of the details might be lost (Fig. 10A). For the second test field (Fig 10B) it can be observed that for rounded shapes (upper part) the shape is similar to the shape approximated from the TLS point cloud. Unfortunately, similar to the first test site (Fig. 10A), the shape was not correctly approximated and maximal deviation are about 20 mm. In the case of the third (decorative stucco frame), which is characterized with the simple shape the deviations between the cross-sections are the lowest (Fig. 10C). Authors CC BY 4.0 License. 672

7 Figure 10. The example of cross-section. Red - Xiaomi point cloud, Blue - TLS data. A) figures of winged female sphinxes B) fire place, C) decorative stucco frame 5. CONCLUSIONS As a result of the conducted analysis we feel confident to positively evaluate the possibility of using Xiaomi 4K cameras in the documentation of Cultural Heritage objects, especially architecture and architectural elements. The achieved errors of the finished documentation are less than 1 cm for objects with simple geometry. From the wall flatness analysis it result that for up to 95% of points the distance from the fitted plane is no more than 8 mm. During the process of images gathering it was necessary to provide uniform lightning to the documented surfaces in order to avoid problems during orientation and generation of the point cloud. For more complex shapes, the obtained data was compared with data gathered through TLS. The performed analysis shows that the error in the shape rendering for up to 95% of points in no greater than 9 mm. In the case of the frame of the fireplace, which is characterized by a texture which is difficult to process, as it is smooth and has very little characteristic points, the generated point cloud had the accuracy no greater than 12 mm. When using Xiaomi 4K cameras in the process of documentation it is crucial to remember that the lens is a fisheye type lens. From the conducted experiment it results that the distortion occurring on the image s edges can be correctly removed during the self-calibration process. Additionally, the average point cloud density was 30 points/dm 2. All of the above results justifies the use of Xiaomi 4K cameras in the documentation of Cultural Heritage monuments, despite them being a low-cost device. However, it has to be taken into consideration that with the rising complexity of the documented object the accuracy of the documentation decreases, and that the process of data gathering requires specific conditions to be satisfied, such as a uniform and sufficient lighting. To conclude, we believe, that in some aspects of the process of documentation the Xiaomi 4K can successfully substitute professional devices, and be an efficient low-cost alternative for them. REFERENCES Brown D.C. 1971, Close-Range Camera. Calibration. Photometric Engineering. vol. 37. no. 8, pp Cardenal J., Mata E., Castro P., Delgado J., Hernandez M. A., Prerez J. L., Ramos M., Torres M., 2004, Evaluation of a digital non metric camera (Canon D30) for the photogrammetric recording of historical buildings, International Archives of Photogrammetry and Remote Sensing, Vol.XXXV. Clarke T.A., Fryer J.G, 1998, The development of camera calibration methods and models, The Photogrammetirc Record, 1998, Vol. 16(91), pp Georgopoulos A., Tsakiri M., Ioannidis C., Kakli A., 2004, Large Scale orthophotography Using DTM from Terrestrial Laser Scanning. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Istambul, Turkey, Vol. XXXV, Part B5, pp Georgopoulos, A., Makris, G.N., Dermentzopoulos, A., An alternative method for large scale orthophoto production, Proceedings of CIPA 2005 XX International Symposium, 26 September 01 October, 2005, Torino, Italy. Gianniou P., Georgopoulos A., Tsakiri M., della K., 2007 The documentation of the medieval entrance of the Rhodes fortification complex, XXI International CIPA Symposium 1-6 October 2007, Athens Greece Kraus K., 1997, Photogrammetry Advanced Methods and Applications, Vol 2, Ferd. DummlerVerlag. Kersten T.P., Stallmann D., Tschirschwitz F., Development of a new low-cost indoor mapping system - System design, system calibration and first results. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. - ISPRS Arch. 41, Authors CC BY 4.0 License. 673

8 Markiewicz, J. S., Łapiński, S., Bienkowski, R., and Kaliszewska, A.: The Example Of Using The Xiaomi Cameras In Inventory Of Monumental Objects - First Results, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-2/W8, , , Remondino, F., Spera, M. G., Nocerino, E., Menna, F., Nex. F., State of the art in high density image matching. The Photogrammetric Record, , pp Wang H, Shen S. and Lu X., 2012, Comparison of the Camera Calibration between Photogrammetry and Computer Vision, International Conference on System Science and Engineering Zhang, C., and Yao, W. 2008, "The Comparisons of 3D Analysis Between Photogrammetry and Computer Vision." The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences Vol. 37, pp Authors CC BY 4.0 License. 674