Parallel Lighting and Reflectance Estimation based on Inverse Rendering

Transcription

1 Parallel Lighting and Reflectance Estimation based on Inverse Rendering Tomohiro Mashita Hiroyuki Yasuhara Alexander Plopski Kiyoshi Kiyokawa Haruo Takemura Figure 1: Virtual object overlay examples: The upper row shows the original image. The highlighting varies depending on the viewpoint. The middle row is an overlay with the estimated reflectance and lighting added. The variation of highlighting for virtual object behaves similarly to that of real object. The lower row is an overlay with a virtual sphere added. Our system renders appropriate AR scenery in terms of lighting environment because of the adequate highlight and attached shadow. A BSTRACT Photometric registration is one of the more challenging problems related to augmented reality (AR) because the simultaneous estimations of both lighting and reflectance are especially difficult problems due to large number of parameters and ill-posed problems. As a result, most currently utilized lighting and reflectance estimation mashita@ime.cmc.osaka-u.ac.jp yasuhara@lab.ime.cmc.osaka-u.ac.jp alexander.plopski@lab.ime.cmc.osaka-u.ac.jp kiyo@ime.cmc.osaka-u.ac.jp takemura@ime.cmc.osaka-u.ac.jp methods employ light probes such as mirror spheres, omnidirectional cameras, or require preliminary scanning of the target object. However, these light probe types are not fully suitable for AR systems. In this paper, we introduce an in-situ lighting and reflectance estimation method that does not require specific light probes and/or preliminary scanning. Our method uses images taken from multiple viewpoints while data accumulation and lighting and reflectance estimations run in the background of the primary AR system. As a result, our method requires little manipulations for image collection. We tested our method in simulated environment and simple real environments. Index Terms: H.5.1 [Information interfaces and presentation]: Artificial, Augmented, Virtual Realities ; I.4.8 [Image Processing and Computer Vision]: Photometric registration 102

2 1 INTRODUCTION Photometric registration is one of the more important issues that must be handled by augmented reality (AR) systems that are intended to display photorealistic virtual objects. To achieve photometric registration, various data, including the geometric and photometric environment of the real scene, is required. Furthermore, in the case of an AR system, which shows virtual objects generated from real objects, the reflections properties also should be utilized. One of the more practical methods used to obtain lighting environment information requires the use of an omnidirectional camera or a spherical mirror [1, 2, 3]. However, such devices are not commonly available. This makes them unsuitable for mobile AR systems. It is also possible to estimate lighting and reflectance from an image [4]. However, one of the problematic points of this method type is the abundance of ill-posed problems and the nonlinear optimization of the lighting and reflectance parameters. Therefore, there are few AR systems that can estimate lighting and reflectance in real-time. Therefore, in order to achieve an efficient in-situ and photorealistic AR system, it is necessary to solve the problems related to lighting and reflection estimations. To accomplish this, we developed a system that estimates lighting and reflectance from input images and geometric information while running in the background of an AR system. Since one of our purposes is to separate lighting and reflectance from multiple images, we adopted an approach that minimizes the differences between a real object and its synthesized images. In this paper, we report the use of a dichromatic reflectance model that incorporates the Phong specular reflection model. The other assumptions used for the estimation are the static and white light sources and homogeneous reflectance parameters of an object. In an AR system, the time from the start of the system to the time the user begins the AR experience should be minimized. Therefore, initial pre-processing and object scanning by the user should be minimized as well. Our system can minimize those initial user operation requirements because the system begins to create estimations using data obtained as soon as the user starts experiencing the AR scene. Furthermore, our system shows the current results provided by the estimation process, which runs in background, and renews the input images used for optimization purposes every iteration. Contributions and limitations Contribution 1: No specific light probe or preliminary scanning of target object. In our method, users do not have to prepare specific light probes such as spherical mirrors or omnidirectional cameras, and it is not necessary to make a preliminary scan of a target. Instead, the system selects a key frame and estimates lighting and reflectance parameters using another thread. As a result, there are few requirements from system startup to the time when the user begins the AR experience. Contribution 2: Simultaneous estimation of lighting and reflectance consisting of specular and diffuse. Our method separates lighting and reflectance from observed intensity. Therefore, it is applicable to AR systems that control lighting and reflectance in environment, such as a system for relighting with an estimated object s reflectance or a system for editing an object s color and highlights under a real lighting environment. Limitation 1: Inapplicable to textured objects. Our proposed method assumes homogeneous surface material properties, constant surface color and specularity. Limitation 2: Low expressiveness for lighting and specular reflectance. The lighting model used in this study incorporates the Phong reflectance model and multiple point light sources. While these models are simple and require fewer parameters, the resulting expressiveness of lighting and reflection is somewhat low. 2 RELATED WORKS A number of approaches to achieving photometric registration have been studied previously. Those methods can be grouped as methods that use reflectance information from a specified object [7, 1, 8], methods that use shading from a specified object [9, 10], and methods that do not use specified objects [2, 3, 11, 12, 14, 15, 16]. Our method is categorized as one that does not require the use of a specified object. Therefore, in this section, we will discuss methods that do not use specific objects as light probes. Lalonde et al. [11] proposed a method for estimating lighting environments from a single outdoor image. This method estimates the sun position and visibility by determining the ground, vertical surfaces, and their relation to a convex object. Additionally, Liu and Granier [16] proposed a method to track outdoor lighting variations in which two types of sequentially variable light sources (the sun and sky) are assumed, and the relative intensities of sunlight and sky light are estimated by using a sparse set of planar feature points extracted from each frame. Knecht et al. [13] proposed a reflectance properties estimation method that runs interactive framerate and does not need preprocessing. This method estimates Bidirectional Reflectance Distribution Function (BRDF) using known geometry and lighting environment. Neverova et al. [12] proposed a method estimating the position and color of multiple light sources using an RGB-D camera. Their method is based on the assumption that the surface is dichromatic, reflectance is homogeneous, and a small surface light source is used. The method decomposes an original image into specular shading, diffuse shading, and albedo, and then identifies a lighting condition that minimizes the differences between the original and rendered images. Gruber et al. [14] proposed an AR system that performs shadow in both ways, real-to-virtual and virtual-to-real shadows. This system estimates distant light fields using an RGB-D input image by determining the relationships between the various normal vectors present in a real scene and their intensities. This system assumes that the observed surface is lambertian. Furthermore, to reduce computational costs, the distant light field is approximated by spherical harmonics. Consequently, this system is capable of running dynamic scenes in real-time. Jachnik et al. [15] proposed an AR system that performs realistic reflection by capturing surface light fields [17]. A surface light field is a hemispherical lighting and reflectance property that covers a particular plane. This proposed system extracts specular the component from the captured surface light field, which is then used to create an environmental map estimate. However, to obtain a surface light field, the user is required to scan a target object hemispherically, which restricts the use of this system to static environments. 3 LIGHTING AND REFLECTANCE ESTIMATION 3.1 Problem Setting Lighting Previous AR systems used for estimating lighting environments assume distant light source models because of reduced computational costs and/or an assumption of outdoor use. In the case of a distant light assumption, such systems only estimate the direction of the light sources. However, in an indoor environment, light sources (such as desktop or ceiling lights) are normally close to the subject of the estimation, which means that the light source model should also have a distance property. 103

3 Geometric registration Some methods obtain geometric information in real-time by using a common or RGB-D camera [5, 6, 20]. In our method, we assume that such geometric information, including positions and orientations of a moving camera and the normal vector on the surface of an object, are all known. Reflectance A dichromatic reflectance model, specifically the Phong model [21], is used as the reflectance model in our AR system because of its simplicity and need for fewer parameters. Other dichromatic reflectance models used in AR systems include the Torrance-Sparrow model [22] and the Oren-Nayar model [23]. 3.2 Parameter estimation A lighting estimation is basically achieved by applying the steepest descent method to the error function, which is defined as a square of difference between the real and synthesized images. The acquisition speed for initial values and the minimal number of parameters required for optimization are critical to the accuracy of our system because it is designed to optimize numerous parameters simultaneously. As a result, it was necessary to introduce a new method for determining initial values Initial value estimation Lighting direction estimation The position of a light source consists of distance d L from a particular point in a real environment and direction is based on the normal of a plane (ϕ L,θ L ), where ϕ L and θ L are the azimuth angle and zenith angle, respectively. The range of ϕ L and θ L are [0,2π) and [0,π/2], respectively. To estimate lighting and reflectance, we introduce an intensity map to show a distribution of reflected intensity obtained from multiple views. This intensity map is generated by identifying areas with high levels of reflected light, which are then used to plot light directions. Lighting direction estimates are achieved by clustering the high intensity areas. In our method, we used Leader-based Clustering [18] to produce estimated directions. However, before clustering is performed, the intensity map is transformed to a gray scale image and smoothed with a Gaussian filter. Finally, the reflected direction of the representative point for each cluster is assumed as a lighting direction. Object color An initial value of object color is also estimated from the object s diffuse reflection. There are two approaches to produce a diffuse color estimation. Nishino et al. [19] used the minimum value of the observed intensity as the diffuse color, while Wood et al. [17] used a median value. In our proposal, we adopt a median value for the initial value of object color estimation because Jachnik et al. [15] showed the robust ability of median filters for tracking errors in an AR system. Other parameters For parameters whose initial values are difficult to estimate, values were assigned heuristically. In practice, the light distance is defined based on the height of the ceiling, while the coefficients of the ambient reflection, diffuse reflection, specular reflection, and the shininess parameter of the target material are defined as the center of each range Optimization The number of parameters in the case of m light sources are (10 + 3m), as shown in Table 1. The directions of light sources are not included in the non-linear optimization because we assume that the estimation described in Sec produces a satisfactory level of accuracy. However, if an AR system is to be used in a dynamic lighting environment, the lighting direction estimation should also be included in the optimization process. In this paper, we assume the following lighting and reflectance conditions: No Initial value Estimation Estimation Image, Camera position and orientation Initialized? Yes Optimization Estimation thread Rendering AR Scene Lighting and Reflectance Parameters Figure 2: Flow of Lighting and reflectance estimation Uniformed diffuse color and fixed specular color Using this assumption, we can disregard the diffuse and specular reflectance differences resulting from the geometric difference. The color of the specular reflection is fixed as (O sr,o sg,o sb ) = (1,1,1). White point light sources We assume white point light sources. A white light source assumption is common in lighting environment estimations because light color estimation is basically ill-posed. Using this assumption, the geometric dependent diffuse color and light source color parameters are excluded from the non-linear optimization parameters. Additionally, the specular reflection of the Phong reflection model used in our system does not consider the color of an object. Due to the need to reduce parameter numbers, our system estimates those (7 + m) parameters for optimization. The error function for the optimization is as follows: E = v views x,y pixels λ R,G,B R v (x,y,λ) S v (x,y,λ) 2, (1) v views x,y pixels λ R,G,B 1 where, R v (x,y,λ) is the pixel value at the (x, y) pixel coordinates of vth frame and S v (x,y,λ) is the pixel value of a rendered image with the same geometric information and current parameters. 4 IMPLEMENTATION Our system consists of two processes, lighting and reflectance estimation and scene rendering. Figure 2 shows the flow of our system. Since these processes are handled independently using multiple threads, an AR scene is rendered using the current best parameters obtained from the estimation thread. Our system uses a desktop personal computer (PC) (3.40 GHz Intel(R) Core(TM) i7-2600, nvidia GeForce GTX 460, 8 GByte memory) and a camera (Point Grey Flea3, 640 x 480 pixels, 30 fps). Table 1: Number of parameters Parameter Range Description L i ϕ i [0,2π) θ i [0,π] Direction of m light sources d Li [0, ] Distance of m light sources O dλ [0,1] Diffuse reflection color (λ R,G,B) O sλ [0,1] Specular reflection color (λ R,G,B) k a [0,1] Ambient reflection coefficient k d [0,1] Diffuse reflection coefficient k s [0,1] Specular reflection coefficient n [0,128] Shininess parameter for material 104

4 Residual error (a) Model 1 (b) Model 2 Figure 3: Simulation models Estimation error [deg] Number of viewpoints Estimation error [deg] Model Number of viewpoints Model 2 Figure 4: Relationships between the lighting direction estimation error and the number of viewpoints Our system uses Parallel Tracking And Mapping (PTAM) [20] for the geometric registration. PTAM is also processed independently using another thread. To gather images used for the estimation, our system captures key frames at 10-degree movement intervals. Since the system can begin estimations using a one key frame, a user can begin using the system without scanning an object or waiting for images to accumulate. 5 RESULTS 5.1 Simulation environment A simulation environment is used for trial and evaluation because the estimated parameters can be compared to those parameters used for rendering. Figure 3 shows the models used for our simulation Light source estimations The relationships between the number of viewpoints and the lighting direction estimation error are shown in Fig 4. These results show that the lighting direction estimation error decreases with increases to the number of viewpoints. The variation of residual error from 10 viewpoints is shown in Fig. 5, where Model 2 shown in Fig. 3 is used for this evaluation. The residual error decreases significantly until the 50th iteration. The synthesized images using parameters during the estimation process are shown in Fig. 6, where Fig. 6 (a) is a real image and Figs. (b) to (f) are images synthesized using the parameters included during the optimization process. The average processing time per iteration is sec Estimation of multiple light sources An evaluation of the estimation method used to obtain the number of light sources and their directions was performed in a simulated environment with eight point light sources. The results of estimations produced from 10 and 20 viewpoints are shown in Figs. 7 and 8, where the green and yellow lines and dots indicate ground truth Number of iterations Figure 5: Variation of residual error with a simulation environment and estimated direction, respectively. These results show that while an incorrect estimate was produced when 10 viewpoints were used, the correct number of light sources could be estimated from 20 viewpoints. The accuracy of the light source direction estimations also improved when the number of viewpoints increased. Closeups of 7 (b) and 8 (b) are shown in Fig. 9. Figure 9 (a), the case of 10 viewpoints, shows that there are no data around some of the correct view directions and only two light sources were estimated correctly. In contrast, Fig. 9 (b) shows that the light source direction is estimated correctly. 5.2 Real environment We then demonstrated our method in a real environment. In this test, the system assumes the target object to be used for the estimation is square because the proposed system has not yet been combined with an object shape detection system. The target used was an expanded polystyrene board to which color and shininess was added via spray paint. The user is required to assign four feature points to define a target plane Lighting direction estimation in a real environment An example of the intensity map is shown in Fig. 10. This intensity map, which is a pixel image, was generated from 26 viewpoints, smoothed with a Gaussian filter, and then transformed into a gray scale image. The yellow cross in the figure is the centroid of a cluster of high intensity pixels Optimization Figure 11 shows the variation of residual error using 16 input images. As can be seen in the figure, the residual error greatly decreases until the tenth iteration, and there are few differences in the visual appearance after that point. Figure 12 shows variations to the synthesized images within an optimization. The processing time is 7.46 sec for the tenth iteration, and the average time per iteration is sec Overlaying virtual object We then demonstrated the process of overlaying a virtual object. Our system does not prevent the user from experiencing AR because the estimation process runs in the background. Generally speaking, the lighting and reflectance converge to low difference state within a few seconds. Figure 1 shows the result of a virtual sphere overlay using the estimated parameters. In this figure, we can see that the highlighting for each virtual object varies depending on the viewpoint. The position and spreading of the highlight in the middle row of Fig.1 indicates that the lighting direction estimation is valid. The lower row in Fig. 1 shows that our system renders 105

5 Real image Initial values (a) Light source direction (b) Intensity map Figure 8: Example of estimation of light source direction from 20 viewpoints 5 iterations 10 iterations (a) Close-up of Fig. 7 (b) (b) Close-up of Fig. 8 (b) Figure 9: Close-ups of Figs. 7 (b) and 8 (b) 50 iterations 228 iterations Figure 6: Appearance variation due to iterations (a) Light source direction (b) Intensity map Figure 7: Example of light source direction from 10 viewpoints appropriate AR scenery in terms of lighting environment because of the adequate highlight and attached shadow. 6 DISCUSSION The results of the simulations discussed above confirmed that the intensity maps can be used to estimate the directions of light sources in some cases, but that the estimated result was insufficient to express a real lighting environment in other cases because of the assumption made when using point light sources. There were also cases of unstable light source number estimates and inaccurate lighting directions when numerous light sources were present in a real scene. To address these issues, it will be necessary to improve the clustering algorithm. The processing time for optimization is about seven sec on the condition that the convergence of optimization is defined as the user might be unaware the difference by iteration. Although the processing time required for optimization increases with the number of viewpoints and light sources, we believe that the convergence time is practical because the main AR system shows the virtual object using the current estimation and users do not have to wait for the optimization to complete. To achieve faster and/or more accurate optimization, another reflectance and lighting model or optimization method will be necessary. However, the appropriate processing time and estimation accuracy depends on how the method is applied because the relationships between processing time and accuracy are basically trade-offs. The photoreality achieved using the proposed method is cur- rently limited because the lighting environment is limited to multiple point light sources. To improve the reality level, a distant light field should be adopted and the limitation on homogeneous reflectance parameters should be relaxed. Furthermore, the ability to handle complicated texture reflections needs to be introduced. 7 CONCLUSION In this paper, we presented an online lighting and reflectance estimation method for AR systems. This method estimates the parameters of lighting consisting of multiple point light sources and reflectance via the Phong reflection model. The estimation consists of initial values estimations and non-linear parameter optimizations that minimize the differences between real and synthesized images. The estimation process runs in the background of an AR system uses the current best parameters when rendering virtual objects. We evaluated the performance of our method using a synthesized environment. The result of the evaluations show that the accuracy of lighting direction estimation and the number of light sources increases with the number of input images. Additionally, we implemented our proposal on an AR system to demonstrate its utility in a real environment. The system shows that lighting and reflectance are estimated online and virtual objects are rendered appropriately by using the best current estimation results. Our plans for future work includes applying a more complicated lighting environment, relaxing the limitations related to uniformed diffuse and specular, and the implementation of a system combined with geometric registration method that can ascertain detailed shapes in real-time. ACKNOWLEDGEMENTS This research was funded in part by Grant-in-Aid for Scientific Research (B), # from the Japan Society for the Promotion of Science (JSPS), Japan. REFERENCES [1] M. Kanbara and N. Yokoya: Real-time estimation of light source environment for photorealistic augmented reality. In proceedings of the 17th International Conference on Pattern Recognition (ICPR), volume 2, pages , [2] T. Kakuta, T. Oishi, and K. Ikeuchi: Virtual kawaradera: Fast shadow texture for augmented reality. In Proceedings of the 10th International Society on Virtual Systems and MultiMedia (VSMM), pages , [3] B. T. Nikodỳm: Global illumination computation for augmented reality. Master s thesis, Czech Technical University in Prague,

6 Figure 10: Intensity map example real image initial value Residual error one iteration 5 iterations Number of iterations Figure 11: Variation of residual error with a real environment 10 iterations 100 iterations Figure 12: Synthesized image variations [4] K. Hara and K. Nishino: Variational estimation of inhomogeneous specular reflectance and illumination from a single view. The Journal of the Optical Society of America A, 28(2), pages , [5] R. A. Newcombe and A. J. Davison: Live dense reconstruction with a single moving camera. In Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , [6] R. A. Newcombe, A. J. Davison, S. Izadi, P. Kohli, O. Hilliges, J. Shotton, D. Molyneaux, S. Hodges, D. Kim, and A. Fitzgibbon: Kinectfusion: Real-time dense surface mapping and tracking. In Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages , [7] P. Debevec: Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques (SIGGRAPH), pages , [8] T. Aoto, T. Taketomi, T. Sato, Y. Mukaigawa, and N. Yokoya: Position Estimation of Near Point Light Sources using Clear Hollow Sphere. In Proceedings of 21st IAPR International Conference on Pattern Recognition (ICPR2012), pages , [9] I. Sato, Y. Sato, and K. Ikeuchi: Illumination distribution from brightness in shadows: Adaptive estimation of illumination distribution with unknown reflectance properties in shadow regions. In Proceedings of the 7th IEEE International Conference on Computer Vision (ICCV), volume 2, pages , [10] T. Takai, S. Iino, A. Maki, and T. Matsuyama: 3-D Lighting Environment Estimation with Shading and Shadows Image and Geometry Processing for 3-D Cinematography (R. Ronfard & G. Taubin, Eds.), Geometry and Computing 5, Springer, (2010). [11] J. F. Lalonde, A. A. Efros, and S. G. Narasimhan: Estimating the natural illumination conditions from a single outdoor image. International Journal of Computer Vision, pages 1 23, [12] N. Neverova, D. Muselet, and A. Trémeau: Lighting estimation in indoor environments from low-quality images. In Proceedings of the 12th European Conference on Computer Vision (ECCV), pages , [13] M. Knecht, G. Tanzmeister, C. Traxler, and M. Wimmer Interactive BRDF Estimation for Mixed-Reality Applications Journal of International Conference on Computer Graphics, Visualization and Computer Vision (WSCG):pp (2012) [14] L. Gruber, T. Richter-Trummer, and D. Schmalstieg: Real-time photometric registration from arbitrary geometry. In Proceedings of the 11th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), [15] J. Jachnik, R. A. Newcombe, and A. J. Davison: Real-time surface light-field capture for augmentation of planar specular surfaces. In Proceedings of the 11th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), [16] Y. Liu and X. Granier: Online tracking of outdoor lighting variations for augmented reality with moving cameras. IEEE Transactions on Visualization and Computer Graphics (TVCG), volume 18, number 4, pages , [17] D. N. Wood, D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D. H. Salesin, and W. Stuetzle: Surface light fields for 3D photography. In Proceedings of the 27th annual conference on computer graphics and interactive techniques (SIGGRAPH), pages , [18] A. Kirmse, T. Udeshi, P. Bellver, and J. Shuma: Extracting patterns from location history. In Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (GIS), pages , [19] K. Nishino, Z. Zhang, and K. Ikeuchi: Determining reflectance parameters and illumination distribution from a sparse set of images for view-dependent image synthesis. In Proceedings of the 8th IEEE International Conference on Computer Vision (ICCV), volume 1, pages , [20] G. Klein and D. Murray: Parallel tracking and mapping for small AR workspaces. In Proceedings of the 6th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages , [21] B. T. Phong: Illumination for computer generated pictures. Communications of ACM, volume 18, number 6, pages , [22] K. E. Torrance and E. M. Sparrow: Theory for off-specular reflection from roughened surfaces. The Journal of the Optical Society of America, volume 57, number 9, pages , [23] M. Oren and S.K. Nayar: Generalization of Lambert s Reflectance Model. In Proceedings of the 21st annual conference on computer graphics and interactive techniques (SIGGRAPH), pages ,