Information and Communication Engineering July 2nd 2010 Fast Spectral Reflectance Recovery using DLP Projector Sato Laboratory D2, 48-097403, Shuai Han Abstract We present a practical approach to fast spectral reflectance recovery by exploiting the unique colorforming mechanism of Digital Light Processing (DLP) projectors. On the basis of this approach, an imaging system that can take spectral measurements at 100 Hz was built. In our system, a DLP projector is used as a light source. Scenes appearance under the projector s irradiation is captured by a high-speed camera. Illumination spectra of the captured frames are recovered by using a white board as a calibration target. Express spectral reflectance as a linear combination of a set of basis functions, our system can reconstruct the spectral reflectance of scene points based on the captured frames and recovered illuminations. We verified the recovery accuracy of our system by using Macbeth ColorChecker. Moreover, effectiveness of the system is demonstrated by spectral relighting of a dynamic scene. 1 Introduction The amount of light reflected on an object s surface varies for different wavelengths. The ratio of the spectral intensity of reflected light to incident light is known as spectral reflectance. It is an intrinsic characteristic of an object, independent of illuminations and imaging sensors. Therefore, spectral reflectance offers direct descriptions about objects which are useful for a wide range of vision tasks, such as color constancy, object recognition, tracking etc. For spectral reflectance recovery, several methods have been proposed. Maloney used a common RGB camera to recover spectral reflectance under ambient illumination [1]. This method is limited by low recovery accuracy due to its RGB 3-channel measurement. To acquire measurements of more than 3 channels, some works acquire the changes in appearance by attaching filters to a light source to modulate illumination [2] or sequentially placing a set of band-pass filters in front of a monochromatic camera to produce a multi-channel camera [3]. These methods sacrifice temporal resolution for spectral resolution, and as a result, they are unsuitable for dynamic scenes. To increase temporal resolution, a special light source is built [4]. It works synchronously with an RGB camera for spectral measurement at 30 fps. Since such self-made light sources, as well as the controller for synchronization are not easily available, it requires some effort to build a similar system. What is more, using these light sources, the size of working volume is restricted by their non-uniform irradiation. What we seek next is a practical system for fast spectral reflectance recovery built on easily available devices. In this work, we exploit the unique color-forming mechanism of Digital Light Processing (DLP) projectors and apply it for spectral measurements. DLP projectors use color wheels to produce desired light. The color wheels are composed of several color segments, and the light that gets through these segments has different spectral distributions. In other words, DLP projectors provide several spectrally distinct illuminations. When the color wheels rotate quickly, light emitted from the DLP projectors switches among these illuminations rapidly. Making use of this switch, we built an imaging system for spectral reflectance recovery. In the system, a DLP projector is used as a light source, and a high-speed camera is adopted to capture scenes appearance under the projector s irradiation. A standard diffuse white board is placed in workspace to recover illumination spectra of captured frames. In order to reduce the number of required measurements for accurate spectral reflectance recovery, we express spectral reflectance as a linear combination of a limited number of spectral bases as done in previous studies [5, 6]. Using this linear model, the spectral reflectance can be reconstructed by the captured frames and the recovered illuminations. The contributions of this work are summarized below. Dense temporal spectral measurement: Our system is capable of taking spectral measurements at 100 Hz. This enables measurement of 1
fast-moving objects, and the recovered results are degraded little by motion blur. Easily built imaging system: Considering that high-speed cameras are becoming readily available on end-use market and no synchronization between the projector and the camera is required, our system can be easily replicated by others. Furthermore, with the DLP projectors as light sources, irradiation uniformity in the whole projection plane can be guaranteed, so the calibrations are simple and the working volume is large. This paper is organized as follows. Section 2 gives a brief review of related works. Section 3 presents our imaging system and its use for spectral reflectance recovery. section 4 verifies its accuracy. Section 5 shows relighting results of a static scene and a moving object. We conclude with section 6 in the end. 2 Related work Spectral reflectance can be recovered under passive illumination. Maloney and Wandell used color constancy and an RGB camera for spectral reflectance recovery [1], but the accuracy of their method is low due to RGB 3-channel measurement. For accurate results, Tominaga put a set of band-pass filters in front of a monochromatic camera to measure more than 3 channels [3]. However, it tradeoffs temporal resolution for spectral resolution, thus is not suitable for dynamic scenes. Other existing methods for spectral reflectance recovery rely on active illumination. DiCalro and Wandell recovered spectral reflectance as an intermediate result [7], but its accuracy is limited by expressing surface reflectance as a combination of 3 spectral bases. To recover spectral reflectance with high accuracy, D Zmura proposed a method using distinct illuminations[8], but the author only showed results using synthetic data, and it is unclear how well the proposed method works for real scenes. Cui, Yoo, and Ben-Ezra proposed an algorithm for selecting an optimized set of wide-band filters and built a multi-illumination system [2]. They attached selected filters to a light source, and used it as additional light source for spectral reflectance recovery under ambient illumination. This method works well for static scenes. However, switching among different illuminations is time-consuming, so the system is not applicable for moving objects. To measure dynamic scenes, Park et al. built an imaging system based on multiplexed illumination [4]. They focused on combinations of different LEDs and built special LED panels to capture 30 fps multi-spectral videos. However, their system requires specially built LED panels and synchronization between the LED panels and a camera. Accordingly, their system is not easily available. Moreover, using these self-made LED panels, irradiation uniformity can be guaranteed only in a small area, so the working volume is quite limited. Our work is also related to DLP-based active vision. Nayar, Branzoi, and Boult implemented a programmable imaging system using a modified DLP projector-camera pair [9]. Users can control radiometric and geometric characteristics of captured images by this system. Narasimhan et al. exploited the temporal dithering of DLP projectors for a wide range of applications [10]. Zhang and Huang used fast illumination variation of a DLP projector for real-time 3D shape measurement [11]. These three works only utilize the fast alternation between the on and off statuses of the digital micromirror device in a DLP projector; color information is disregarded. In contrast, our work is the first to recover spectral reflectance using a DLP projector. 3 Spectral Reflectance Recovery 3.1 Overview Image brightness is related to three factors: incident light, the scene and camera. This relationship can be expressed as I m,n = s(λ)c m (λ)l n (λ)dλ, (1) where I m,n is the intensity of a scene point in captured frames, s(λ) is the spectral reflectance of that point, c m (λ) is the spectral response function of the camera at the mth color channel and l n (λ) is the spectrum of the nth illumination. The goal of this work is to recover spectral reflectance s(λ) in a visible range [400 700nm]. From Eq. 1, we can see that a large set of spectrally distinct measurements are required if we want to recover s(λ) with high spectral resolution. For instance, recovery of s(λ) every 10 nm in [400 700nm] leads to 31 unknowns, and thus requires at least 31 measurements. However, capture of so many spectrally distinct measurements is not only complicated but also time-consuming, so it is not suitable for fast spectral reflectance recovery. If we reduce the spectral resolution, such as from 10 nm to 50 nm, the number of required measurements can be decreased. However, the accuracy of the recovered results is not sufficient. 2
Figure 1: System prototype. For recovery without sacrificing spectral resolution, we express spectral reflectance as a combination of a limited number of basis functions. Several linear models [5, 6] and a nonlinear model [12] have been built by principal component analysis [13] or other tools (see Ref. [14] for a review). With regard to how many bases are required for accurate reconstruction, different works have different conclusions [5, 6, 15, 16, 17]. We adopt an 8-dimension linear model for spectral reflectance derived from [6] on account of its high reconstruction accuracy. In this way, spectral reflectance can be reconstructed by estimating eight coefficients. The linear model can be expressed as s(λ) = 8 β j Φ j (λ), (2) where Φ j (λ)(j = 1, 2,.., 8)is the jth spectral basis from Ref. [6] (spectral resolution:10nm), β j is the corresponding coefficient. Substituting Eq.2 into Eq.1 I m,n = 8 β j Φ j (λ)c m (λ)l n (λ)dλ (3) Suppose the camera has a linear intensity response, Eq. 3 can be transformed into I m,n = 8 β j Φ j (λ)c m (λ)l n (λ)dλ (4) In our work, we first estimate β j from observed I m,n. Then, spectral reflectance s(λ) is reconstructed by substituting β j into Eq. 2. As shown in Fig. 1, our imaging system is composed of a one-chip DLP projector (PLUS TM U2-1130), 500 fps RGB camera (PointGrey TM Lightning), and a standard diffuse white board (labsphere TM SRT-99). The camera has a linear intensity response and its spectral response function, c m (λ) (m = 1, 2, 3), is measured Figure 2: Camera s spectral response function for RGB 3 channels. by a spectrometer and a monochromator. The measured spectral response function is shown in Fig. 2. Using our imaging system, we recover spectral reflectance in the follow steps. 1. Image acquisition: Scene s appearance under the projector s irradiation, I m,n, is acquired by using the high-speed camera. Every five consecutive images are used as one measurement for spectral reflectance recovery. (Section 3.2) 2. Illumination recovery: Illumination spectra, l n (λ), changes from frame to frame. We use the white board as a calibration target to recover illumination of captured frames. (Section 3.3) 3. Spectral reflectance reconstruction: Based on the 8-dimensional linear model, spectral reflectance, s(λ), can be reconstructed from the acquired images and recovered illuminations. (Section 3.4) In the following, we explain each of these steps in detail. 3.2 Image acquisition Different from other kinds of projectors, DLP projectors use a color wheel to produce desired light. The color wheel consists of several color segments, and these segments allows only light in a specific wavelength range to get through. When a color wheel rotates quickly, the emitted light from the DLP projectors changes rapidly. In this work, this temporal variation of light is referred to as color switch. A diagrammatic sketch is shown in Fig. 3. In our system, the DLP projector is equipped with a 3-segment color wheel that rotates at 120 rps, so color switch occurs at 360 Hz (3 120) when we input (255, 255, 255). Human eyes, or common video cameras that work at low rates (24 30 Hz), cannot detect the color switch. In our work, the 500 fps camera is adopted to take videos of scenes under the projector s irradiation. During one rotation of the color wheel, the high-speed camera can capture 3
Figure 3: Color switch caused by rotation of color wheel. 4.17 frames. To cover all 3 spectrally distinct illuminations, we use 5 consecutive frames as one measurement for spectral reflectance recovery. Fig. 4 shows us one measurement about Macbeth ColorChecker. We can see that the scene s appearance clearly changes under the color switch of the DLP projector. 3.3 Illumination recovery Our system does not require synchronization between the projector and the camera. Due to the asynchronism, illuminations changes from frame to frame. Thus we need to recovery illumination spectrum, l n (λ), of every frame. In this section, we describe a method by using the white board as a calibration target. As mentioned above, light that gets through different segments on color wheels has distinct spectral distributions. If we use these spectral distributions as illumination bases, light emitted from the DLP projectors in a certain period can be expressed by a linear combination of these bases. In our system, since the 3 segments of the color wheel corresponding to RGB color filters, we can get these 3 distinct illuminations by inputting the projector (255, 0, 0), (0, 255, 0), and (0, 0, 255) respectively. Their spectra, measured by a spectrometer, are shown in Fig. 5. Therefore, l n (λ) in our system can be expressed as l n (λ) = 3 α n,k b k (λ), subject to α n,k > 0, (5) k=1 where b k (λ) is the spectrum of the kth illumination basis, α n,k is the corresponding coefficient. Different combinations of coefficients enable the DLP projector to produce millions of kinds of light with different spectral distributions. From Eq.1 and Eq.5, the brightness of a surface point on the white board is Figure 4: One measurement of Macbeth ColorChecker and corresponding illumination spectra. Top row is 5 frames captured sequentially by 500 fps camera in 1/100s. Bottom row is recovered illuminations of these frames. Im,n w = s w (λ)c m (λ) = k=1 3 α n,k b k (λ)dλ k=1 3 α n,k b k (λ)s w (λ)c m (λ)dλ, (6) where I w m,n is the intensity of that point, and s w (λ) means its spectral reflectance. By letting B k,m represent the intensity of the white board at the mth channel under the kth illumination basis B k,m = b k (λ)s w (λ)c m (λ)dλ (k = 1, 2, 3), (7) Eq. 6 can be rewritten as I w m,n = 3 α n,k B k,m (8) k=1 Using the white board s appearance under 3 distinct illuminations of the projector, B k,m (k = 4
Figure 5: Spectra of RGB illuminations of the DLP projector. 1, 2, 3) can be directly measured. We only need to measure them once in advance. From Eq. 8, we see that the intensity of a surface point on the white board under illumination l n (λ) is a linear combination of its intensities under three illumination bases I w I w 1,n B 1,1 B 2,1 B 3,1 α n,1 n = I 2,n w = B 1,2 B 2,2 B 3,2 α n,2 = B w α n, I3,n w B 1,3 B 2,3 B 3,3 α n,3 (9) where I w n represents the RGB value of a surface point on the white board under the nth illuminations, B w is a matrix consists of B k,m (k = 1, 2, 3 m = 1, 2, 3), α n is the corresponding coefficient vector. In principle, α n can be easily calculated by α n = (B w ) 1 I w n. However, due to the noise, α n,k (k = 1, 2, 3) may be negative sometimes. This conflicts with the non-negative constraint of Eq. 5. Thus, we solve α n as a non-negative least squares problem: α n = argmin α n I w n B w α n 2 subject to α n,k 0 (k = 1, 2, 3) (10) by using calculated α n, illumination spectra, l n (λ), can be reconstructed by Eq. 5. 3.4 Spectral reflectance reconstruction By Eq. 4: I m,n = 8 β j Φj (λ)c m (λ)l n (λ)dλ, the intensity of a scene point can be computed. l n (λ) is recovered in the previous section, thus the integral in this equation can be represented as known coefficients: f j,m,n = Φ j (λ)c m (λ)l n (λ)dλ. One measurement that contains five consecutive frames can be written in matrix form as I = F β, (11) where I is a 15 1 vector (15 measurements: RGB 3 channels 5 frames), F is a 15 8 matrix (15 measurements 8 spectral bases), and β is an 8 1 coefficient vector. If β is estimated from I, spectral reflectance s(λ) can be reconstructed as: s(λ) = 8 β j Φ j (λ). this way, the problem of spectral reflectance recovery can be solved through 8 coefficients estimation. The DLP projector in our system has 3 spectrally distinct illuminations, and the high-speed camera provides a 3-channel measurement under each illumination. In total, we can obtain 3 3, i.e. 9 effective channels. Thus, the problem of estimating eight coefficients becomes over-determined. However, by using of the least squares solution of Eq. 11, the reconstructed spectral reflectance does not always satisfy the non-negative constraint and the solutions tend to be unstable. Therefore, we adopt a constrained minimization method proposed in Ref. [4]. We use the first derivative of the spectral reflectance respective to λ as a constraint: [ β = argmin I F β 2 +c smooth s(λ) ] β λ 2 subject to Φβ 0 for all λ, (12) where c smooth is a parameter for constraint term. Φ is the 8 spectral bases matrix. 4 Accuracy Evaluation In this section, we evaluate the accuracy of our system by Macbeth ColorChecker. In the system, every 5 consecutive frames captured by the 500 fps camera is used as one measurement. Thus, spectral measurements are taken at 100 Hz. The color wheel rotates at 120 rps. Due to the asynchronism between the DLP projector and the camera, illuminations of the frames captured at different time have different spectra. The accuracy of the recovered results would be affected by the illumination variation. In this section, we are going to evaluate both the spectral accuracy and temporal accuracy of our system. To evaluate spectral accuracy, we sequentially took 200 measurements (1000 frames) of a static 24- clip Macbeth ColorChecker. For every clip, we set c smooth to 50 and reconstructed its spectrum based on the measurements; then, the root mean square (RMS) error of the 200 reconstructed results was calculated; after that, we computed the maximum, mean, and minimum of the 200 RMS error values. The results for all 24 clips are shown in Fig. 6. We In 5
Figure 6: Maximum, mean and minimum RMS error of 24 clips of Macbeth ColorChecker for 200 measurements. Figure 7: Average value of RMS error of 24 clips for 200 measurements. For every 5 measurements, color wheel rotates 6 rounds. Thus, a pattern can be seen. can see that our system can accurately recover spectral reflectance of many clips, such as foliage. For some other clips, though the recovered results have comparatively bigger error because inaccuracies occur at the red range, they are still acceptable. For every clip, its maximum RMS error does not deviate a lot from the minimum one. And the biggest mean RMS error of all 24 clips is less than 0.11. These results demonstrate that our system can recover spectral reflectance with reasonable spectral accuracy. Next, we evaluate the temporal accuracy of our system. We reused the 200 measurements taken in previous test. For every measurement, we reconstructed the spectral reflectance of all 24 clips; then, the RMS error of the 24 reconstructed results was calculated; after that, we computed the average value of the 24 RMS error values, and use it as the criterion to evaluate each measurement. Results for all 200 measurements are shown in Fig. 7. The average value fluctuates in a narrow band (0.047, 0.06) which verifies the temporal accuracy of our system. Figure 8: Recovered spectral reflectance of some clips on Macbeth ColorChecker by the measurement shown in Fig. 4. Ground truth: red lines; recovered: black lines. 5 Image and Video Relighting We used the spectral reflectance recovered by our method for spectral relighting of a static scene as well as a moving object. To ensure strong and spatial uniformly distributed light, an LCD projector (EPSON TM ELP-735) was used as the light source for relighting. Spectral distributions of its white, red, green, and blue were measured by a spectrometer. 5.1 Image relighting We set a static scene with fruits, vegetables and small statues. Five consecutive frames of the scene were captured by our imaging system. Using them as one measurement, the spectral reflectance of scene points was recovered pixel by pixel. Then the scene was spectrally relit by Eq. 1 with known illumination spectra of the LCD projector. A comparison between the relit results and the real captured images is shown in Fig. 9. From the comparison, we can see that the computed results are very similar to the ground truth. It reveals the accuracy our system. 6
Illumination Computed Ground Truth evaluation, the accuracy and the robustness of our system have been verified. Moreover, our system is built on easily available devices, and the excellent optical design of DLP projectors guarantees simple calibrations and large working volume. Therefore, it can be concluded that our system is practical and robust for spectral reflectance recovery of fastmoving objects. References Figure 9: Comparison between relit results and captured images of a static scene under distinct illuminations of the LCD projector. 5.2 Real video relighting Our system works at 100 Hz, so it is capable of measuring fast-moving objects. By our system, spectral measurements of a manipulated toy were captured continuously. For every measurement, spectral reflectance of scene points was reconstructed. Based on the recovered data, toy s movements were spectrally relit under a variety of illuminations. From Fig. 10, we can see that the computed results look natural. In the bottom of Fig. 10, a relit result is shown in the middle. It was computed on the basis of the spectral data recovered by our system. The left is a real image captured by the high-speed camera under the LCD projector s irradiation. A synthesized result to simulate captured image by a 30 fps camera is shown on the right side. Through comparisons, we can see that the relit result resembles the real captured image, and it is not degraded by the motion blur which is obvious in the synthesized result. These comparisons show the robustness of our system to artifacts caused by motion. Thus, it can be said that our system is suitable for fast-moving objects. 6 Conclusion In this work, we exploited the unique color-forming mechanism of DLP projectors. Making use of this mechanism, an imaging system for fast spectral reflectance recovery was built. This system is capable of taking measurements as fast as 100 Hz. Every measurement consists of a set of sequentially captured images. For each set, spectral reflectance of scene points can be recovered. Through intensive [1] Laurence T. Maloney and Brain A. Wandell. Color constancy: a method for recovering surface spectral reflectance. Journal of the Optical Society of America A, 3(1):29 33, 1986. [2] Cui Chi, Hyunjin Yoo, and Moshe Ben- Ezra. Multi-spectral imaging by optimized wide band illumination. International Journal on Computer Vision, 86:140 151, Jan 2010. [3] Shoji Tominaga. Multichannel vision system for estimating surface and illumination functions. Journal of the Optical Society of America A, 13(11):2163 2173, 1996. [4] J. Park, M. Lee, M. D. Grossberg, and S. K. Nayar. Multispectral Imaging Using Multiplexed Illumination. In IEEE International Conference on Computer Vision, Oct 2007. [5] J.Cohen. Dependency of the spectral reflectance curves of the munsell color chips. Psychon. Sci, 1:369 370, 1964. [6] J. Hallikainen J. Parkkinen and T. Jaaskelainen. Characteristic spectra of munsell colors. Journal of the Optical Society of America A, 6(2):318 322, 1989. [7] Feng Xiao Jeffrey M. DiCarlo and Brian A Wandell. Illuminating illumination. In Proc. Ninth Color Imaging Conference, pages 27 34, 2000. [8] Michael D Zmura. Color constancy: surface color from changing illumination. Journal of the Optical Society of America A, 9(3):490 493, 1992. [9] T. E. Boult S. K. Nayar, V. Branzoi. Programmable imaging: Towards a flexible camera. International Journal on Computer Vision, 70:7 22, Oct 2006. [10] S. J. Koppal S.G Narasimhan and S. Yamazaki. Temporal dithering of illumination for fast active vision. In Proc.European Conference Computer Vision, Oct 2008. 7
Captured image (500 Hz) Relit result(100 Hz) Synthesized image(30 Hz) Figure 10: Top two rows: relit results of fast-moving toy. Continuous movements through very different illuminations are shown. Bottom row: left is image captured by 500 fps camera, middle is relit result, right is synthesized result to simulate captured image by a 30 fps camera. Recovered result by our system is degraded little by motion blur. [11] Song Zhang and Peisen Huang. Highresolution, real-time 3d shape acquisition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops, volume 3, pages 28 37, 2004. [12] Feng Xiao Jeffrey M. DiCarlo and Brian A Wandell. Spectral estimation theory: beyond linear but before bayesian. Journal of the Optical Society of America A, 20(7):1261 1270, 2003. [13] Roy S. Berns. Di-Yuan Tzeng. A review of principal component analysis and its applications to color technology. Color Research And Application, 30(2):84 98, 2006. necessary? Journal of the Optical Society of America A, 9(4):507 515, 1992. [16] Thomas W. Cronin Chuan-Chin Chiao and Daniel Osorio. Color signals in natural scenes: characteristics of reflectance spectra and effects of natural illuminants. Journal of the Optical Society of America A, 17(2):218 224, 2000. [17] Laurence T. Maloney. Evaluation of linear models of surface spectral reflectance with small numbers of parameters. Journal of the Optical Society of America A, 3(10):1673 1683, 1986. [14] and Timo Jskelinen. Oili Kohonen, Jussi Parkkinen. Databases for spectral color science. Color Research And Application, 31(5):381 390, 2006. [15] James L. Dannemiller. Spectral reflectance of natural objects: how many basis functions are 8