Estimating the Cook-Torrance BRDF Parameters In-Vivo from Laparoscopic Images

Transcription

1 Estimating the Cook-Torrance BRDF Parameters In-Vivo from Laparoscopic Images Abed Malti and Adrien Bartoli ~ab} ALCoV-ISIT, UMR 6284 CNRS/Université d Auvergne, 28 place Henri Dunant, Clermont-Ferrand, France Abstract. SfS (Shape-from-Shading) and view synthesis systems generally assume a diffuse reflection model of the in-vivo tissues where the light is equally reflected in all directions. In other words, they approximate the tissue s BRDF (Bidirectional Reflectance Distribution Function) by the Lambertian model. This is however a coarse assumption since most tissues cast specularities. We propose a method to estimate the reflectance properties of tissues from invivo laparoscopic images. We use the Cook-Torrance BRDF model in order to take into account both diffuse and specular properties of the tissues. Our method estimates online both the BRDF parameters of the observed organ and the light model of the laparoscope. Such an estimation requires the knowledge of the 3D shape and some geometric priors on the light source. For these reasons, our estimation method relies on two assumptions: firstly, that the tissues undergoes rigid motion when the surgeon only explores it, and secondly that laparoscope s light is colinear to the viewing direction in a neighborhood ring around the specular regions. The first assumption allows us to estimate the 3D shape of the organ using for instance classic RSfM (Rigid Structure-from-Motion). The second assumption allows us to estimate the BRDF parameters. The determination of the 3D shape and the BRDF parameters allows us to assign a light direction for each pixel of the image. Experimental results compare the performance of our joint BRDF-and-light estimation method with the widely used Lambertian model. This validation uses both ex-vivo and real in-vivo datasets. It reveals a substantial improvement on SfS 3D reconstruction. 1 Introduction Over the last decade, computer aided laparosurgery has attracted extensive research and interest. It consists in improving the practitioner s perception of the intra-operative environment [9]. In the context of Augmented Reality for Computer Assisted Intervention (AE-CAI), 3D sensing offers a synthetic controllable view-point and is one of the major possible improvements to the current technology. In order to supplement standard laparoscopes with this type of facility, it is of critical importance to recover depth accurately from images. Laparoscopic images exhibit strong variability conditioned on the optical properties and pose of the laparoscope. The reflection of the incident light by the tissues and its appearance according to the view point is described by the so-called BRDF. Estimating this function is likely to improve higher-level tasks such as new view synthesis from SfS. This can help the surgeon to see the organs from different point of views. These views can be substantially augmented with surgery tools. The estimation of a BRDF in-vivo has however not been addressed yet in the context of laparosurgery. Most of the rendering and SfS methods in computer aided surgery

2 use a simple Lambertian reflectance. This model suits diffuse matte surfaces without specularities. However, on the one hand, living tissues tend to be more specular than diffusive and on the other hand, the BRDF parameters may not be shared by the different organs. It is unfortunately not possible to pre-estimate BRDF parameters ex-vivo and use them in-vivo since the reflectance properties change due for instance to moist. We propose an online estimation method for the BRDF of a specific organ. The estimated BRDF follows the analytical model proposed by Cook and Torrance [4]. This BRDF is better adapted than the Lambertian model since it models the surface as a distribution of specular microfacets accounting for color constancy (the amount of variation in chromaticity among a batch of similar tissues). In order to have a better representation of the light-tissue interaction we propose a light model adapted to the context of laparoscopy: our light model uses a direction vector per pixel. The overall light vector flow is fitted to a cubic B-spline parameterized in the 2D pixel space of the image. Experimental comparisons between our proposed method and a Lambertian model using ex-vivo and in-vivo organs reveal substantial improvement on SfS 3D reconstructions. Paper organization. Section 2 presents background and related work. Section 3 reviews the Cook-Torance reflectance model. Section 4 presents our method for BRDF estimation. Section 5 presents our method for light calibration. Section 6 describes the steps of the implementation of our method. Section 7 reports experimental results. Finally section 8 concludes. Our notation will be introduced throughout the paper. image views... image processing neighbors of specular set... tracked features... non-specular set... (ii) BRDF (i) RSfM (iii) Light 3D Shape Cook-Torrance Model (ρ,f, σ) image width BRDF Parameters image height Light directions Fig. 1. Stages of our method for online joint BRDF and light estimation: (i) 3D shape reconstruction of the organ based on RSfM, (ii) assuming light direction is collinear to viewing direction in regions neighboring the specular parts, we use a set of images to obtain a global estimation of the BRDF parameters, and (iii) using the BRDF parameters, we estimate the light direction for each pixel of the laparoscope s image. The specific organ is here a uterus. The Cook-Torrance BRDF model is described in section 3.

3 2 Background and Related Work Describing and modeling surface reflection has been a field of investigation in computer vision and computer graphics for decades. The BRDF describes how the incident light is modulated by a surface patch. It is a positive function of four angular dimensions and that be writtenf(ω i,ω o ), whereω i andω o are unit vectors in the hemisphere centered about the patch normal. These vectors are, respectively, directions of incident light and viewing direction (more precisely reflected light). The BRDF provides a complete description of appearance for optically-thick surfaces for which mutual illumination and sub-surface scattering are negligible [8]. We assume that these conditions hold in our context. One can measure the BRDF of a planar material by sampling the double-hemisphere of input and output (ω i and ω o ) directions with a gonioreflectometer. Since this is extremely slow, and since a slight loss of accuracy is often acceptable for vision and graphics applications, a number of camera-based alternatives have been proposed. Phong [1] proposed a reflectance model for computer graphics that was a linear combination of specular and diffuse reflection. The specular component was spread out around the specular direction by using a cosine function raised to a power. Subsequently, Blinn [2] used similar ideas together with a specular reflection which accounts for the offspecular peaks that occur when the incident light is quasi-orthogonal to the surface normal. Other analytical models have been popular for their compact representation [13, 4]. Recent approaches represent BRDFs as combinations of a set of basis functions [1, 11, 12]. However, the number of bases needs to be kept large to account for viewing and lighting variability and to maintain high frequency details. In the medical context few works addressed the problem of BRDF estimation for tissues. A patient-specific BRDF estimation method for bronchoscopy was proposed in [3]. This method is not well-adapted to laparoscopy mainly for two reasons: (i) it assumes that the light direction is colinear with the viewing direction, which cannot be realistic at every pixel of a laparoscopic image as experimentally demonstrated and (ii) the BRDF model is not adapted to surfaces with a non-negligible specular component. The main contribution of our paper is a new method to describe the joint BRDF-light estimation in endoscopy. Our method features (a) the Cook-Torrance BRDF model, (b) light calibration and (c) runs online, relying on 3D organs shape reconstruction using RSfM. Our working assumptions are as follows: (a) the tissues undergo rigid motion when the surgeon explores it, (b) the light direction is collinear to the viewpoint only in regions around specularities. Our proposed method has three main stages illustrated in figure 1: (i) 3D shape reconstruction of the organ based on RSfM, (ii) a global estimation of the BRDF parameters, and (iii) light direction estimation for each pixel of the laparoscope s image. 3 BRDF Modeling: The Cook-Torrance Model The Cook-Torrance model [4] was developed based on geometrical optics and is considered as one of the most physically plausible model. The basis of this model is a reflectance definition that relates the brightness of an object to the intensity and size of

4 single image V H N L Cook-Torrance Fig. 2. Left: for a Lambertian model the image intensity at a surface point depends on the laparoscope s light source direction and the surface normal at that point. Right: local geometry of reflection as described by Cook and Torrance [4]. In this case, the image intensity at a surface point depends also on the viewing direction V. N is the surface normal at incident point, L is light direction and H is the bisector of L andv. each visible light source. Thus, at a given image pixel q, the predicted image intensity Î depends on three vectors: the shape normaln, the viewing directionv and the light direction L (see figure 2 for a local geometry representation of the reflection). It is given by: Î = ρ π (N L) }{{} diffuse reflectance + F D π (N L) (N V) } {{ } specular reflectance The diffuse reflectance is assumed to be Lambertian and ρ is the diffuse albedo. The constant Fresnel coefficient F represents the refractive index of the tissue. The facet slope distribution function D represents the fraction of the facets that are oriented in the direction ofh, the bisector oflandv. Cook and Torrance [4] used the Beckmann distribution function: ( ) 1 D = σ 2 cos 2 α exp tan2 α σ 2 (2) where α is the angle between N and H. The parameter σ is the root mean square of the microfacets and represents the surface roughness. Some surfaces have two or more scales of roughness, and can be modeled by using more than one distribution function. In these cases, D is a weighted sum of the distribution functions, i.e., (1) D = p w p D(σ p ) (3) where σ j is the surface roughness of the jth distribution and the sum of the weights is one [4]. 4 Estimating the Cook-Torrance BRDF s Parameters In-Vivo For estimating a BRDF, reflections are first measured under various viewing and illumination angles. The data are then usually fitted to an analytical model using leastsquares non-linear minimization [7]. Nonlinear BRDFs that include multiple Gaussianlike functions such as the Cook-Torrance model generally induce a large number of local minima in the cost function.

5 Given an organ of known shape and a light of known direction, we want to estimate the parameters ρ, F and σ of the Cook-Torrance model from a set of laparoscopic images. With known object shape, we have N. For a rigid laparoscope where the light is rigidly mounted on the tip, we cannot consider the light directionl as being known at every pixel of the image. However, we can assume that the light direction is approximately colinear to the surface normal in the vicinity of specularities. For convenience, we denote as S the set of specular pixels in the image, S the set of non-specular pixels which are close neighbors to pixels in S and S the set of all non-specular pixels. The viewing direction is constant and coincides with the camera view axis. In this case, we can seek for the BRDF parameters by writing the parametric prediction equation of the Cook-Torrance model for a given pixel q of the image: ( Î(ρ,F,σ) = aρ +bf 1 σ 2 exp( c ) σ 2), ρ, F, σ (4) where ρ = I s ρ, F = I s F, a = N L π, b = 1 π(n L)(N V)cos 4 α, c = tan2 α. I s is the light intensity and if unknown, ρ and F are estimated up to a scale factor. For notation convenience, the dependence with respect to q is not explicitly displayed in the equations. Thus a first approach to estimate the BRDF parameters ρ, F and σ would be to minimize the RGB error between the predicted intensityî and measured intensityi: ( ( (ρ, F, σ) = argmin I aρ +bf 1 (ρ, F, σ) σ 2 exp( c )) 2 σ 2) (5) S Problem (5) is non-convex and non-linear with respect to σ. Moreover, it is difficult to find a decent initialization for the triplet (ρ, F, σ) to reach a global minimum with non-linear iterative optimization method. However, if we assumeσ = σ as being known, the problem of finding (ρ, F ) turns convex and the optimal values for ρ and F can be found using Second Order Cone Programming [14] to the restricted problem: min t s.t. I ( aρ +bf 1 exp( c )) σ 2 σ t, in S, 2 ρ, F (6) In order to have a global estimate of σ, we embed problem (6) in a global BnB (Branch-and-Bound) estimator as follows: (a) a bound interval of admissible σ values is subdivided into several non-overlapping sub-intervals (except at the boundaries). (b) At the center of each sub-interval we test the feasibility of problem (6) and discard all infeasible sub-intervals. (c) The restricted convex optimization (6) is solved for the center values of each interval. (d) The interval which gives the best solution of problem (6) regarding to the minimum RGB error (5) is kept. We repeat steps (a, b,c, d) until the length of the interval becomes small enough, then we keep the last center σ and the corresponding solutions ρ and F of (6). In our implementation, we experimentally set the initial interval to [1 5, 1] and the threshold interval-length to 1 3. The SOCP problem (6) is solved with YALMIP-toolbox [6] which allows us to detect infeasible

6 sub-intervals and solve the feasible ones. Once we determine the BRDF parameters, we take advantage of the fact that the laparoscope s light source is rigidly attached to the lens tip to estimate a slant and tilt direction 3-vector of the light at each pixel of the laparoscopic image. 5 Calibrating Light In-Vivo We assume that the light direction is colinear to the laparoscope s view point in S to estimate the reflection parameters of the tissues. However this assumption does not hold for all pixels in S. In order to compensate the error that may accumulate from the RGB error of the image intensity, we re-estimate the light direction in S. In the specular region S the light direction is assumed to be perfectly colinear to the viewing direction. In our model, the light direction is parameterized with respect to the image pixelq: L = ( sin(φ)cos(θ), sin(φ)sin(θ), cos(φ) ) forq S L = (,, 1 ), forq S (7) withθthe tilt angle of the light direction in the image plan with respect to the view axis, θ [ π,π]. φ is the slant of the light direction in the image plan with respect to the view axis, φ [, π 2 ]. Again, the dependency with respect to q is not explicitly written for notation convenience. The light direction is estimated by minimizing the RGB error with respect to θ andφ: ( ( N L L = argmin I ρ F 1 σ exp ( ) )) 2 c + 2 σ 2 L π π(n L)(N V)cos 4 (8) (α) S Problem (8) is non-linear with respect to the light parameters and is solved using Levenberg-Marquardt. To have a decent initialization, the light direction is determined by propagating the estimation from the boundary conditionl = ( 1 ) at the specular set S toward its close neighbors. Thus from close neighbors to close neighbors, we determine the light direction at each pixel of the image where a shape normal information is available. Finally, the estimated sample direction angles θ and φ are fitted to a cubic B-spline [5]. 6 Implementation and Calibration Steps We use a set of M laparoscopic images from the exploration of an organ s tissues. In summary, the BRDF and light estimation steps are as follows: Step 1 A geometric shape of the organ s tissues is reconstructed using the M views with RSfM. Step 2 For each view, specularities on the reconstructed shape are detected via combined saturation and lightness thresholding (we use thresholds of saturation.95 and lightness.95 respectively to detect specular pixels) and correspondences between specular pixels and shape points is established. All the specular pixels are gathered into the set S and all the corresponding shape points are labelled with their normals to the surfacen.

7 Step 3 For each view, we determine the set of pixels which are close neighbors to S. In our implementation, we choose the pixels which belong to a ring centered at the specular pixels with 3 pixels of maximum distance from the the specular frontiers. All these neighbor-to-specular pixels are gathered into the set S and all the corresponding shape points are labelled with their normals to the surfacen. Step 4 Using the set S we estimate the BRDF parametersσ,f andρ as described in section 4. The Fresnel parameter F and the diffuse parameter ρ are estimated up to scale since we assume that the light intensity is unknown. Step 5 We calibrate the light direction all over the image plane at pixel resolution as described in section 5. 7 Experimental Results To validate our BRDF-light estimation method, we proceed to a comparison with the classical Lambertian model on three criteria: (i) the RGB error of the M calibration images, (ii) the RGB error on a set ofm test images and (iii) the 3D reconstruction using SfS. The RGB error is computed as the difference between the measured image intensity I by the laparoscope and the predicted image intensity by the considered model (either Cook-Torrance or Lambertian). The SfS 3D reconstruction using the Lambertian model uses the algorithm of Tsai and Shah [15]. In the absence of an SfS algorithm using the Cook-Torrance model, we use the estimated normals by Tsai and Shah algorithm as initial estimates that we iteratively refine as: N = argmin N ( I ( N L ρ + π F 1 σ 2 exp ( c σ 2 ) π(n L)(N V)cos 4 (α) )) 2 (9) In order to upgrade the SfS reconstruction to the metric scale, we use the scale size of the shape. This allows us to compute the 3D error of reconstruction as the norm of difference between the depths of the SfS reconstruction and the ground-truth. 7.1 Ex-Vivo Datasets with Ground-Truth In order to acquire a real ex-vivo dataset we setup an appropriate framework (see figure 3) where we use two laparoscopes fixed through two trocars mounted on a pelvitrainer. The two laparoscopes are mounted to two PointGrey Flea2 color cameras with two c- mounts. The two cameras are synchronized at 15 fps with a resolution of pixels. A light source is mounted to one of the laparoscopes. This laparoscope will be used to evaluate our method for online BRDF and light calibration. Thanks to this setup we can build accurate ground-truth 3D models of ex-vivo organs with stereo views. A set of5 images of a lamb s lungs are acquired with our setup. We use3 images to estimate the BRDF and the light model as described in section 6. We use the set of2 remaining views to compare our method with the Lambertian model by estimating RGB error and SfS 3D reconstructions (see figure 4). As can be seen, our estimation method fits the measured image intensities more precisely and has lower 3D reconstruction error. Figure 5 highlights these quantitative results by showing qualitative improvements in 3D reconstruction when using our method. As can be seen, it better recovers the surface in the presence of specularities.

8 2 laparoscopes surgery tool camera + adapter pelvitrainer light source Fig. 3. Experimental setup to acquire real ex-vivo datasets. Two Pointgrey cameras are synchronized to obtain reference ground-truth data using stereo-views. During the experiment the laparoscope s light source is the unique source of light in the setup. 3 4 [-255] RGB Lambertian Cook-Torrance [-255] RGB Lambertian Cook-Torrance 3D error [mm] SfS(Lambertian) SfS(Cook-Torrance) Fig. 4. Ex-vivo lungs datasets. From left to right: RGB error on calibration images, RGB error on test images and 3D error of the SfS reconstruction (The bounding box surrounding the Lungs has a volume of [mm 3 ]. This volume is used to recover the scale size of the SfS reconstructions.). As can be seen, our estimation method fits the measured image intensities more precisely and has lower reconstruction error. Image Ground-Truth SfS(Lambertian) SfS(Cook-Torrance) Fig. 5. Ex-vivo lungs dataset. SfS reconstructions: as can be seen, the estimated Cook-Torrance model fits better to surface in the presence of specularities.

9 7.2 In-Vivo Datasets The experiment on real data we propose is the 3D reconstruction of a uterus from an in-vivo sequence of 3 images acquired using a monocular Karl Storz laparoscope running at 25 fps with a resolution of pixels. The 3D shape of the uterus is generated during the laparosurgery exploration step of the inside body. We use a set of 2 images to estimate the BRDF and the light model as described in section 6. We use a set of 1 remaining images to evaluate our method with classic Lambertian model by estimating RGB error and SfS 3D reconstructions (see figure 6). It can be observed that with our method, we can have a shift of 2 RGB levels (over 255 range values) while the Lambertian model can reach up to 6 RGB levels of errors. In terms of 3D reconstruction errors, it can be seen that taking into account the specular component of the BRDF brings a clear performance gain above the Lambertian model. Figure 7 highlights these quantitative results by showing qualitative improvements in 3D reconstruction when using our method. As can be seen, it better recovers the surface in the presence of specularities. 5 6 [-255] RGB [-255] RGB D error [mm] Lambertian Cook-Torrance -1 Lambertian Cook-Torrance SfS(Lambertian) SfS(Cook-Torrance) Fig. 6. In-vivo uterus datasets. From left to right: RGB error, RGB error on test images and 3D error of the SfS reconstruction (The bounding box surrounding the uterus has a volume of [mm 3 ] which is consistent with an average human uterus. This volume is used to recover the scale size of the SfS reconstructions.). As can be seen, our estimation method fits better the measured image intensities and has better reconstruction error (here the SfS reconstructions were compared to RSfM 3D reconstructions). 8 Conclusion In this paper, we have presented a method for the in-vivo acquisition of the Cook- Torrance BRDF parameters and calibration of the light direction rigidly mounted on the tip of a laparoscope. We experimentally showed (quantitatively and qualitatively) that this model suits better the physical reflectance of tissues than the Lambertian model. SfS 3D reconstruction gives promising results as an application of our method in the context of 3D laparoscopy. In future work we are planning to reduce the computation time to reach 3D real time requirements. References 1. R. Basri and D. W. Jacobs. Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(2): , February 23.

10 Image Rigid SfM SfS (Lambertian) SfS (Cook-Torrance) Fig. 7. In-vivo uterus dataset. SfS reconstruction. As can be seen, our estimation method reconstructs better the surface in regions which are close to specular areas. 2. J. F. Blinn. Models of light reflection for computer synthesized pictures. In SIGGRAPH, A. J. Chung, F. Deligianni, P. Shah, A. Wells, and G. Yang. Patient-specific bronchoscopy visualization through brdf estimation and disocclusion correction. IEEE TMI, 25(4):53 513, R. L. Cook and K. E. Torrance. A reflectance model for computer graphics. In SIGGRAPH, P. Dierckx. Curve and surface fitting with splines. Oxford University Press, J. Lofberg. Yalmip : A toolbox for modeling and optimization in MATLAB. In Proceedings of the CACSD Conference, A. Ngan, F. Durand, and W. Matusik. Experimental analysis of brdf models. In Proceedings of the Eurographics Symposium on Rendering, F. Nicodemus, J. Richmond, J. Hsia, I. Ginsberg, and T. Limperis. Radiometry. chapter Geometrical considerations and nomenclature for reflectance. Jones and Bartlett Publishers, Inc., S. Nicolau, X. Pennec, L. Soler, and N. Ayache. A complete augmented reality guidance system for liver punctures: First clinical evaluation. In MICCAI, B. T. Phong. Illumination for computer generated pictures. Commun. ACM, 18: , R. Ramamoorthi and P. Hanrahan. Frequency space environment map rendering. In SIG- GRAPH, I. Sato, T. Okabe, Y. Sato, and K. Ikeuchi. Appearance sampling for obtaining a set of basis images for variable illumination. In ICCV, H. Xiao, K. Torrance, F. Sillion, and D. Greenberg. A comprehensive physical model for light reflection. In SIGGRAPH, C. Yu, Y. Seo, and S. W. Lee. Global optimization for estimating a brdf with multiple specular lobes. In CVPR, Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. Shape from shading: A survey. PAMI, 21(8):69 76, 1999.