Available online at ScienceDirect. Procedia Computer Science 67 (2015 )

Transcription

1 Available online at ScienceDirect Procedia Computer Science 67 (2015 ) th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Infoexclusion (DSAI 2015) Utilizing Multispectral Scanning and Augmented Reality for Enhancement and Visualization of the Wooden Sculpture Restoration Process. Zuzana Berger Haladová a, Robert Szemzö b,tomáškovačovský a,c,ján Žižka c a Faculty of Mathematics, Physics and Informatics, Comenius University, Mlynská Dolina, Bratislava , Slovakia b Academy of Fine Arts and Design, Hviezdoslavovo nám. 18, Bratislava , Slovakia c Photoneo Ltd., Jamnického 3, Bratislava , Slovakia Abstract Nowadays the digitization of the cultural heritage artefacts utilizing 3D scanning (with RGB or multispectral textures) is broadly used. In our project AREST (Augmented reality and reconstruction in the restoration process) we want to create a full pipeline for producing 3D models of artefacts (wooden statues) in different stages of restoration and utilize them to present the restoration process in the comprehensive way to a broad public. To present the process, we propose a device called Virtual Vitrine which will combine the commercial passive stereo display with view dependent stereo and gesture interaction. An essential part of the project is also an acquisition of multispectral images (textures) of artefacts that can be analysed by professional conservators. c 2015 The Authors. Published by Elsevier by Elsevier B.V. B.V. This is an open access article under the CC BY-NC-ND license ( Peer-review under responsibility of organizing committee of the 6th International Conference on Software Development and Peer-review Technologies under for responsibility Enhancing Accessibility of organizing and committee Fighting of Info-exclusion the 6th International (DSAI Conference 2015). on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI 2015) Keywords: Multispectral scanning, 3D reconstruction, Augmented reality, View dependent stereo, Cultural Heritage, Restoration; 1. Introduction In the area of restoration/conservation the key issue is to closely and accurately examine and understand the nature of the artefact to be restored. Since 90 s the usage of digital imaging techniques became an important part of the examination process. The information that can be extracted from 3D scans or images capturing nonvisible parts of the electromagnetic spectrum may help to uncover the hidden layers of the examined artefacts. On the other hand these techniques can be utilized not only to extend the knowledge of professionals but also to provide a sneak peak on the conservation process to the broader public. In this work in progress paper we want to propose the goals and recent Zuzana Berger Haladová. Tel.: address: haladova@fmph.uniba.sk The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license ( Peer-review under responsibility of organizing committee of the 6th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI 2015) doi: /j.procs

2 Zuzana Berger Haladová et al. / Procedia Computer Science 67 ( 2015 ) results of the project AREST-Augmented reality and reconstruction in the restoration process. The project has two main goals: Create a structured light scanner capable of producing 3D models of artefacts together with multispectral textures in different stages of restoration process. Create the Virtual Vitrine- an interactive view dependent stereo display to present the process of restoration. To achieve these goals we will build upon our previous projects SMISS 1 and Virtualizer where the structured light scanner capable of full 3D reconstruction was constructed. This paper is organized as follows. In the second section we will present the previous works in the area of multispectral reconstruction and presentation of the cultural heritage. In the third section we will propose the AREST pipeline. Fourth section will be focused on the scanning process with Virtualizer scanner and its extension to multispectral scanning. Fifth section will describe the Virtual Vitrine concept. In the following sections we will conclude the paper, sketch the tasks of the future work and give acknowledgements. 2. Previous works A rise in development and usability of imaging spectrometry devices (multispectral and hyperspectral devices) in various research areas was connected to growing availability and performance of the digital imaging sensors during last two decades. Various categories of devices are employed. They may be classified according to principle of filtering/dispersion of radiation wavelengths into four groups: Devices using mounted optical interference filters (the filter set is usually mounted on a motorized filter wheel). For example: the multispectral imaging methods used for identification of paint layer colorants in oil paintings 2. Devices utilising crystal tunable filters (LCTF) or acustic-optical tunable filters(aotf) 3. Solutions employing controlled light sources with tunable spectrum of emitted radiation like DLP projector or LED light tunable source 4, 5, 6. Prism grating prism (PGP) spectrometer coupled with ccd camera allowing high spectral resolution 7, 8. Motivations for composition of devices combining 3D spatial and multispectral acquisition are apparent only sufficient amount of data provided by complex digitisation of spatial and radiometric properties of artefacts may lead to effective computer data processing. Development of devices of similar attributes became over the past decade a subject of several projects at various research institutions. In the previously published works there is a great variety of used acquisition systems. The 3D structured light acquisition system created in order to provide a realistic virtual representation of heritage objects exposition was proposed in 9. In 10 the implementation of BDRF estimation for correction of texture mapping of glossy surfaces or proper rendering of surface reflectance was described. In 11 authors employed multispectral camera or thermal camera acquisition of wide surfaces using photogrammetric tracking set for estimation of position and orientation of the system. 3D imaging spectrometry was used for identification of colourants, for example for archaeological artefact and wall paintings analysis 12 or ancient pottery 13. In the area of the virtual restoration of archival documents a combination of multispectral acquisition, 3D imaging and digital image processing were used for improvement of readability, aesthetic properties and to bring the document back to its original appearance 14. Despite few recent achievements 15, 7 it is possible to pronounce that identification of pigment content of mixed paint layers (prevailing phenomenon in traditional painting) is disputable. We consider automatic application of pigment mapping and identification based on spectral reflectivity of their surfaces as unrealistic for most of authentic surfaces of polychrome sculptures. Usually they are not sufficiently cleaned, the various varnish layers may occlude the sensed spectra. Paint layers as well as varnishes are usually modified by their natural or artificial oxidation causing radical shift on their appearance which is not evenly dispersed on their surfaces. It should be remarked that a longer wavelengths (SWIR, NIR) of radiation may achieve a deeper transmittance of subject surfaces 15. Although polychromy of the traditional wooden sculptures is usually noticeably thicker than it was usual for painted surfaces of pictorial art, the contamination of recorded reflectance value from deeper material layers may be not

3 342 Zuzana Berger Haladova et al. / Procedia Computer Science 67 (2015) Fig. 1. Various polychrome surfaces were usually employed for decoration of wooden sculptures. Later interventions like retouches, re-paintings, damages, degradation or losses of polychrome layers may occlude their original appearance. excluded. The painted areas, although they may look like that, are usually not homogenic, they are containing various coloured particles spread in uneven thickness. Visible range spectral measurements of the thin paint layer may bring information of spectral reflectivity of their surfaces. On the other hand infrared spectral range is probably going to refer to measured values of underpainting, or primer that was laid beneath them (the final measurement is not referring to the same layer anymore). We assume that correct definition and mapping of reflectance and fluorescence 16 surface characteristics is a complex phenomena therefore the Flowchart interpretation method 17 is closer to conception of the proposed acquisition system. The multispectral texture acquired in the scanning phase of AREST will be also used to obtain segmentation data based on main sensed differences of paint layers, varnished, unvarnished areas, gilded surfaces, later interventions (like retouches, re-paintings), damages, degradation signs or losses of original polychrome layers. Examples of polychrome wooden sculpture details can be seen in figure AREST pipeline The AREST project can be divided in two main parts: the scanning phase (including postprocessing) and the presentation phase. In the figure 2 the scheme of the process is depicted. The input of the whole process is the artefact (in our case wooden polychrome sculpture) which will be sequentially scanned in different stages of the restoration process. The output of the system is the triangulated mesh model of the object with multispectral texture per every scanned stage. The number of the scanned models per one restored artefact depends on the complexity of the restoration and will be decided by restorer before the start of the process. An example of three stages of the restoration can be seen on figure 6. These models can be than registered together and displayed in the Virtual Vitrine or stored and analysed by professional restorers. More details on the scanning and presentation section will be described in the following sections. Another important part of the proposed pipeline is the analysis of the recorded data by professional conservators. 4. Scanning The first step of our process is the scanning phase which is carried out by the modified Virtualizer scanner capable of producing the complete, high resolution point-clouds of artefacts. The core of the Virtualizer scanner is the structured

4 343 Zuzana Berger Haladova et al. / Procedia Computer Science 67 (2015) $57()$&7 &UHDWH SRLQW FORXG XVLQJ VWUXFWXUHG OLJKW VFDQQHU $FTXLUH PXOWL VSHFWUDO WH[WXUHV,5 5 * % 89 5HSHDW IRU YLHZSRLQWV 0DS WKH WH[WXUH RQ WKH SRLQW FORXG 0DS WKH WH[WXUH RQ WKH SRLQW FORXG 0HUJH SRLQW FORXGV EDVHG RQ WKH UHIHUHQFH SDWWHUQ 3UHVHQWDWLRQ SKDVH 3UHVHQW WKH PRGHOV WR SXEOLF XVLQJ 9LUWXDO 9LWULQH (QKDQFH UHJLVWUDWLRQ XVLQJ,&3 DQG FUHDWH PHVK ' 0RGHO 3RVWSURFHVVLQJ SKDVH 6FDQQLQJ SKDVH 0DQXDOO\ UHJLVWHU ' PRGHOV RI DOO SKDVHV RI UHVWRUDWLRQ $QDO\VH GDWD E\ SURIHVVLRQDO FRQVHUYDWRUV 5HSHDW DIWHU HYHU\ PDMRU UHVWRUDWLRQ VWHS XQWLO UHVWRUHG Fig. 2. The scheme of the AREST project. light optical 3D scanner SMISS 1. The system uses projector-camera setup to reconstruct the surface geometry of target figure using triangulation principle. The Gray and sinusoidal coded patterns are used to solve correspondence problem with sub-pixel accuracy. The scanning system also incorporates the method of High Dynamic Range scanning using projector weight maps. The intensity of projector lighting is weighted per pixel, so that surfaces with higher reflectance receive less illumination. This allows us to capture Higher Dynamic Range scenes with just constant increase of scanning time for weight map capture and computation. The creation of the object texture, the corresponding color values for every reconstructed point are discussed in the following subsection. The Virtualizer scanner consists of SMISS scanner head mounted on a motorized arm, which allows vertical positioning of the sensor. The artefact is placed on a motorized glass table for automatic object rotation. These two axes allow us to position the scanning head on arbitrary point of a sphere with a predefined radius with respect to a center of an object. In the postprocessing phase we can create a complete 3D cloud by merging partial scans from different viewpoints. In our project we will use the fixed position of motorized arm (which will be separately positioned for every scanned artefact) and 16 different uniformly distributed angles of the rotating motorized glass table. The pose of individual scans is obtained from capturing designed pose-estimation marker field of multiple circles with binary coded positions (see figure 3). Fig. 3. (a) Photograph from the scanning process; (b) Two putti embedded on clouds- the sculptural object from 18. century- the untextured model obtained by the structured light acquisition system. The object is laid on custom designed marker field and in every scan multiple circles of the field are localized and pose estimation is computed from image to world space correspondence and the known calibration parameters (world coordinates are encoded as two 16 bit floats), so that the partial point clouds share the marker fields coordinate system. After all necessary partial scans are captured, the ICP (Iterative closest point 18 ) algorithm is used for more precise alignment of individual point clouds. Then there is the possibility to store the created pointcloud, or triangulate it further to create a mesh and/or run the hole filling procedure 1.

5 344 Zuzana Berger Haladová et al. / Procedia Computer Science 67 ( 2015 ) Fig. 4. Comparison of three different textures mapped on the point cloud of painted testing sample created by modified Virtualizer system: (a) Visible light spectrum; (b) Infrared reflectography; (c) False colour infrared image Multispectral extension As described before SMISS scanner consist of the projector-camera setup. The camera used in the system is Point Grey FL3-U3-32S2M/M-CS monochromatic camera (without the infrared cut-off filter). RGB values for reconstructed points were in the SMISS system achieved using sequential projection of R, G and B color (by projector) on to the object surface. As far as the relation between camera pixels and 3D points is known from the reconstruction, RGB values can be assigned based on the values of three subsequent monochromatic images (lighten by R, G and B). There are two motivations for the creation of the multispectral texture. First one is, that IR and UV induced visible fluorescence imaging plays an important role in the examination of the object by professional restorers because it may posses the otherwise invisible information about variety of properties, mainly the inconsistency of chemical composition in different but similar appearing surfaces. On the other hand to accurately present the color of the cultural heritage object (e.g. in Virtual Vitrine) we have to have in mind that a recorded color image is both device and illuminant dependent. Although we can estimate the color-accurate image using conventional trichromatic sensors the image recorded with higher spectral resolution will produce a better estimation of the surface reflectance. To achieve multispectral images we have decided to use the spectral multiplexing approach and modify the existed projector-camera system. The modification process was divided in two phases. In the first (naive) phase we extended the existing solution with two channels capturing the information from the NIR and UVA part of the electromagnetic spectrum. The two light sources emitting electromagnetic radiaton spectra with NIR (narrow band emitting LED) and UVA (externally filtered Black light tube) wavelengths respectively. To create the corresponding images (UV and IR) the eliminating of the surrounding (global) illumination was necessary. That was achieved by recording one image with surrounding illumination (N image) and than consecutively two images with NIR and UVA switched on. Final images were achieved by extracting the N image from the IR and UV images. This method does not necessitate the usage of optical filters and can be plug in to the Virtualizer scanning pipeline. The lights were mounted on the motorized arm close to the projector and synchronized with the scanning process via programmable plug. In this phase we were able to acquire following textures: 3 channel RGB, 1 channel Infrared reflectography (IR) and ultraviolet reflectography (UV). For a comparison of different textures created in this phase see figure 4. The problem with the developed solution is that created IR and UV images have significantly more noise compared to images produced using direct filtration of the spectrum. Also the recorded red channel partially consist of the NIR values produced by the projector. We have decided to overcome these limitations. The next step planned within the AREST system will be the extension of the optical system using motorized wheel with color filters. The main idea is to use 7 filters (IR long pass, UV pass, and 5 bandpass filters uniformly splitting the visible spectrum) and one free position. The acquired data will be stored as a hyperspectral cube. The radiance values will be achieved from hyperspectral values (reflectance pixel values) using chosen illuminant spectrum (corresponding to the illuminant used in the showcase installation). The radiance values will be converted to CIE XYZ using CIE 1931 colour matching functions. Then XYZ values will be converted to RGB and displayed in the Virtual Vitrine. For more details of the process, see 19.

6 Zuzana Berger Haladová et al. / Procedia Computer Science 67 ( 2015 ) To capture the UV induced visible fluorescence properties (giving an important characteristics utilized in the restoration process) we will also use the externally filtered UVA radiation emitter together with R, G and B bandpass filters to create special texture if needed. 5. Presentation One of the main goals of the AREST project is to present and popularize the restoration process in the broad public. To achieve this, we have decided to create a device called Virtual Vitrine. Vitrine will be installed near the real showcase with the already conservated artefact and will allow users to explore the phases of the artefact restoration. The Vitrine will combine the commercial passive stereo display with the view dependent stereo and gesture interaction to provide the user with an immersive experience View dependent stereo To create similarity of the displayed virtual artefact inside the Virtual Vitrine to the appearance of the real sculpture, we have decided to simulate the effect of motion parallax using the view dependent stereo principle 20. The implemented prototype of the system can be seen in figure 5. The system for tracking of the user consists of two Nintendo Wii remote controllers mounted on the metal construction and the polarized glasses (paired with the display) with 2 infrared LED diodes mounted on sites of the glasses. Nintendo Wii remote controllers communicate with computer via bluetooth and have sensors (cameras) capable of determining the center of the diode with resolution 1024x768 and frequency 100 Hz. The principle of computing of the 3D position of the diodes with the described system is identical to a standard stereovision with 2 cameras. The calibration of the system was done in several steps. First we calibrate the intrinsic and extrinsic parameters of the two cameras mounted on the frame. Calibration pattern consist of a square (of a known side) with 4 LED diodes in the corners. Since there are only 4 points captured in every frame, several frames with different pattern position are needed. In this phase the 3D position of the LEDs in new frames are known with respect to coordinate system of one camera. However to implement the view dependent stereo we need to have the coordinates of diodes in the coordinate system of the display. To achieve this, we used the two step calibration with the fixed mirror and the described calibration pattern. First, we estimated the coordinates of the mirror plane in 3D by positioning of the pattern on the mirror surface. Second, we determined the position of the display plane using the reflection of the pattern from the mirror (position is known from the first step). An important part of the immersion to the Virtual Vitrine is the illumination of the virtual object. This can be simulated utilizing the multispectral texture which will be produced for every artefact (and its restoration stage) and the model of the illuminant in the real showcase. Fig. 5. (a) The scheme of the Virtual Vitrine device in exposition with the real showcase with artefact; (b) Example of the functionality of our view dependent system created for the Virtual reality exhibition.

7 346 Zuzana Berger Haladová et al. / Procedia Computer Science 67 ( 2015 ) Interaction with the Vitrine To present the process of the restoration we decided to allow the user to navigate the virtual representation on the time axis switching between scans acquired in different stages of the restoration process. There were efforts to display this process in the past using photographs of the stages which already helps to better understand the phases of the process (for an example see figure 6(a)). To avoid the distraction of the immersed user we decided to prefer gesture navigation to other types of navigation (touch screen, mouse, audio control). The timeline will be displayed on the bottom part of the screen and allow user to switch between different stages by shifting the cursor on the timeline using his hand. To track the motion of the hand we have decided to use the Kinect depth sensor with the SDK which has the skeleton tracking already implemented. The information about the position of the corresponding skeleton joints will be used to recognize the palm position and its movement and will translate the timeline cursor accordingly. Fig. 6. (a) Wooden sculpture of Madonna from Kurov (eastern Slovakia) second half of 14. century. An example of the three phases of past restoration process. Using 3D Multispectral acquisition system, each intervention could be documented, preserving past identity of the object to be unveiled by the reverse-restoration axis. Photographed by O. Csütörtöky, 1986; (b) Polychrome wooden sculpture of St. Sebastian- from Kežmarok (Eastern Slovakia), end of the 15. century. Visualization of underpainted veins visible only in the infrared spectrum images. 6. Conclusion In this work in progress paper, we have proposed the main goals and bases of the project AREST- Augmented reality and reconstruction in the restoration process. In the field of multispectral 3D scanning we have described the existing system Virtualizer which is capable of producing complete 3D models and proposed its modification to multispectral scanning device. The Virtualizer system was build upon the SMISS scanner which was proved to have average accuracy of 0,5 mm, measured by scanning a flat ceramic table in distance of 800 mm. The full scanning (16 viewpoints) of one object takes approximately 10 minutes. For more details about the parameters see 1. The second proposed goal is the creation of the device for presentation of restoration process to the broad public. We have proposed the device called Virtual Vitrine which combines passive commercial stereo display, view dependent stereo and gesture interaction. The constructed prototype view dependent stereo system was described. 7. Future work In the future work we would like to complete all the proposed modifications of existing devices and systems. We are planning to scan the process of restoration of the wooden polychrome sculptures using 5 subsequent scans in different restoration phases. Then will be the conservated object presented aside the Virtual Vitrine in cooperation with lab.sng (the platform of the Slovak national gallery). Using the Vitrine, visitors of the gallery will be able to explore the sculpture in different stages of restoration. Afterwards, we would like to create a user study to validate if the proposed presentation concept helped visitors to explore and understand the process of the restoration of displayed Cultural Heritage object.

8 Zuzana Berger Haladová et al. / Procedia Computer Science 67 ( 2015 ) Acknowledgements This work was supported by the operational program ASFEU project COMENIANA - metódy a prostriedky digitalizácie a prezentácie 3D objektov kultúrneho dedičstva, ITMS: , which is co-financed from resources of the European regional development fund. The project AREST was also partially funded by Etalent grant by Nadácia Tatrabanky. References 1. Kovacovsky, T.,. Hdr smissfast high dynamic range 3d scanner. CESCG, TU Wien 2012;: Zhao, Y., Berns, R.S., Taplin, L.A., Coddington, J.. An investigation of multispectral imaging for the mapping of pigments in paintings. In: Electronic Imaging International Society for Optics and Photonics; 2008, p Mansfield, J.R., Attas, M., Majzels, C., Cloutis, E., Collins, C., Mantsch, H.H.. Near infrared spectroscopic reflectance imaging: a new tool in art conservation. Vibrational Spectroscopy 2002;28(1): Shrestha, R., Hardeberg, J.Y.. Multispectral imaging using led illumination and an rgb camera. In: Color and Imaging Conference; vol Society for Imaging Science and Technology; 2013, p France, F.G.. Advanced spectral imaging for noninvasive microanalysis of cultural heritage materials: Review of application to documents in the us library of congress. Applied spectroscopy 2011;65(6): Lee, H., Kim, M.H.. Building a two-way hyperspectral imaging system with liquid crystal tunable filters. In: Image and Signal Processing. Springer; 2014, p Delaney, J.K., Zeibel, J.G., Thoury, M., Littleton, R., Palmer, M., Morales, K.M., et al. Visible and infrared imaging spectroscopy of picasso s harlequin musician: Mapping and identification of artist materials in situ. Applied spectroscopy 2010;64(6): Cucci, C., Casini, A., Picollo, M., Poggesi, M., Stefani, L.. A hyper-spectral scanner for high quality image spectroscopy: Digital documentation and spectroscopic characterization of polychrome surfaces. In: ART11-10th International Conference on Non-destructive Investigations and Microanalysis for the Diagnostics and Conservation of Cultural and Environmental Heritage. 2011,. 9. Mansouri, A., Lathuiliere, A., Marzani, F.S., Voisin, Y., Gouton, P.. Toward a 3d multispectral scanner: an application to multimedia. IEEE multimedia 2007;14(1): Maczkowski, G., Sitnik, R., Krzesłowski, J.. Integrated digitization method for cultural heritage objects. Photonics Letters of Poland 2012; 4(2):pp Chane, C.S., Schütze, R., Boochs, F., Marzani, F.S.. Registration of 3d and multispectral data for the study of cultural heritage surfaces. Sensors 2013;13(1): Mara, H., Breuckmann, B., Lang-Auinger, C., et al. Multi-spectral high-resolution 3d-acquisition for rapid archaeological documentation and analysis. In: Proc. of 17th European Signal Processing Conference (EUSIPCO 09)(Glasgow, Scotland, United Kingdom, 2009). Citeseer; 2009, p Kim, M.H., Harvey, T.A., Kittle, D.S., Rushmeier, H., Dorsey, J., Prum, R.O., et al. 3d imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics (TOG) 2012;31(4): Bianco, G., Bruno, F., Tonazzini, A., Salerno, E., Savino, P., Zitová, B., et al. A framework for virtual restoration of ancient documents by combination of multispectral and 3d imaging. In: Eurographics Italian Chapter Conference. 2010, p Kubik, M.. Hyperspectral imaging: a new technique for the non-invasive study of artworks. Physical techniques in the study of art, archaeology and cultural heritage 2007;2: Verri, G., Comelli, D., Cather, S., Saunders, D., Piqué, F.. Post-capture data analysis as an aid to the interpretation of ultraviolet-induced fluorescence images. In: Electronic Imaging International Society for Optics and Photonics; 2008, p Cosentino, A.. Identification of pigments by multispectral imaging; a flowchart method. Heritage Science 2014;2(1): Zhang, Z.. Iterative point matching for registration of free-form curves and surfaces. International journal of computer vision 1994; 13(2): Hardeberg, J.Y., Schmitt, F., Brettel, H.. Multispectral color image capture using a liquid crystal tunable filter. Optical engineering 2002; 41(10): Lee, J.C.. Hacking the nintendo wii remote. Pervasive Computing, IEEE 2008;7(3):39 45.