Light source separation from image sequences of oscillating lights

Transcription

1 2014 IEEE 28-th Convention of Electrical and Electronics Engineers in Israel Light source separation from image sequences of oscillating lights Amir Kolaman, Rami Hagege and Hugo Guterman Electrical and Computer Engineering Department Ben Gurion University of the Negev Beer Sheva, Israel Abstract Light has a significant influence on the color of objects in the image. Sometimes scene light is comprised of mixture of several light sources, and this mixture makes it hard to achieve color constancy across the image. Having precise control over the intensity of the light sources at the capturing stage, can enable an easy light source separation by turning on a single light source for each captured frame. In other cases - prior knowledge on the cyclic behavior of the light intensity over time can be used instead, by describing each light source as a set of base functions. This enables the reconstruction of the light sources by means of inner multiplication of the video scene with the base functions. This analysis method assumes that the signal is linear, but this assumption fails when secularities or high illuminance values cause clipping or when low light conditions causes noise to be apparent in the image. By using a cyclic High Dynamic Range (HDR) sampling method the scene becomes linear and good reconstruction results are obtained.two experiments are used to demonstrate the approach. The first shows the decomposition of two oscillating light sources and the second the decomposition of an oscillating light and sunlight. (a) Mixed light source scene I. INTRODUCTION Color is one of the main components used to help us describe our every day lives. Using color helps us describe people, scene, objects and even feelings. In computer vision, color helps the computer the better recognize objects in the scene[1], produce 3D data[2] and more. Light has a big influence on the color of an object in the image [3]. The same object captured by the same camera under different types of illumination may vary in its color measurement values[4]. Color constancy algorithms try to transform the input image into a new image in which all the colors in the scene are independent of the light source illuminating them[5]. This transformation is also referred as White Balance (WB). Many natural scenes have a mixed lighting condition originating from several lights sources, as seen in the example in figure 1(a). Artificial lights, originating from the indoor lighting on the ceiling or walls, have different color temperature (tint) which varies from reddish (tungsten light) to blueish (fluorescent) to green (led). This work proposes a method for separating light sources in the capturing stage, which enables improvement of white balance of a color image, as seen in figure 1(b). (b) Standard white balance Vs. Proposed method Fig. 1. (a) Natural image with two lights sources affecting the image with a different color tint. Notice the table changes in color (zoom-in) from grey on the left to blue on the right. (b) comparison between standard white balance and white balance after light source separation using the proposed method

2 (a) Our sampling method II. RELATED WORK Color constancy is an extensively studied field of research and many algorithms have been proposed [5], most of them focus on static images. This section briefly reviews the most relevant methods to our work, namely those based on sampling video data over time and extracting lights source data from them. Single illuminance estimation from video To the best of our knowledge, there are only 2 works related to color constancy, video sequences and a single light source. In [6] color statistics of a video scene is presented to demonstrate the variance of color over time. In [7] frame averaging of similar video sequences to estimate the scene chromaticity. Multi-illuminance estimation from video The only work found closely related to this work is of Prinet et. al. [8]. Prinet uses the information in the video sequence to recognize two illumination sources and estimate their chroma values. Prinet assumes that the scene has secularities, that the intensity of the light source stays constant over time and that it is evenly distributed over the entire image. She then uses information around the edges to estimate the chroma of two lights in the scene. III. LIGHT SOURCE SEPARATION First a video sequence of Z frames is captured. The image sequences of the scene has a mixture of N light sources. The objective is to separate this sequence into N images illuminated by a single single light source. (b) Our proposed method compared to standard Fig. 2. Sample graphs of on/off light amplitude (marked in blue) represents theoretical intensity of white patch with ideal exposure value. Real sample values of image pixels are marked in colors in image and graph. (a) Proposed sampling method in which light intensity changes sinusoidally, and exposure values changes several times to accurately sample the radiance values of the objects in the scene. (b) General algorithm for improving white balance results on mixed lights scene using light source separation compared to standard white balance algorithm. The main contributions of this work are: 1) Introducing a new method for sampling oscillating light sources from video using High Dynamic Range (HDR) technique, as seen in figure 2(a). 2) Using this sampling method it is possible to improve WB performance, as seen in figure 2(b). The rest of the paper is organized as follows: after a short review on the state of the art color constancy algorithms (Section II), we describe a new method for separating two light sources from a video sequence, assuming that at least one of them is oscillating in time, is presented (Section III). Experiments and analysis are given in Section IV, and conclusions in Section V. A. Naive light source separation Having a precise control over the intensity of light sources in the scene and synchronizing it with the capturing device, enables light source separation (by turning on a single light source in each video frame). This gives N images, where each image has a single light source. The following procedure is performed for each controlled light: 1) turn on a single light source 2) capture a frame The method assumes that for N light sources each light can be turned off separately, and precisely synchronized with the capturing device. This assumption is not always true. For example scenes with a mixture of artificial and natural sun light cannot be controlled. In order to solve this problem the following procedure must be followed: 1) Capture the initial frame with all the lights turned off (except the background light). 2) Capture Z = N 1 frames, where each frame has sunlight and another controlled light turned on. 3) Subtract the first frame from each Z = N 1 frames to get N 1 images with separated light sources. This Naive Light Source Separation assumes that out of N light sources at least N 1 can be precisely controlled and synchronized with capturing device. In most systems, this kind of high precision control is not possible, and light sources are not synchronized to the camera. In these cases the fact that

3 indoor lighting, connected to the power supply, usually change their intensity over time[9] (flickering.), can be used to our advantage. Assuming that out of N light sources in the scene, N 1 light sources change their intensity over time in a linear and cyclic manner, helps us perform the separation. In the next sub section this assumption is used to develop the main idea of light source separation algorithm. B. Separation of sinusoidally varying lights Base functions B n of N dimension, can be used to reconstruct any linear signal over time. An example base function is B n = sin(2π f n) (1) (a) Light intensities graph over time Each light source is modulated by the basis L n = a n B n (2) where B n is described in equation (1) and a n is the amplitude coefficient of the base signal. The composite signal of the light sources is described by: S = N L n (3) n=1 If the base signals are orthogonal and normalized to 1 then B n = 1 (4) < B n, B s > = 0 (5) Extracting the coefficient for each light source is done by â n =< S, B n > (6) Thus reconstructing the nth light source is achieved by ˆ L n = â n B n (7) Intensity of real light source Lr varies between 0 Lr 2 a n. Then, Lr n = a n + a n B n = a n (1 + B n ) (8) ˆLr n = â n (1 + B n ) (9) where Lr, ˆLr represents the real and estimated light source values respectively. In case of N light sources with only N 1 having a cyclic varying intensity the constant light source can be reconstructed by N 1 ˆLc = S Lr n (10) n=1 where Lc represents a light source with a constant intensity over time. (b) Sample points of light intensities graph Fig. 3. (a) A time sample graph of 5 pixel values, where clipping effect can be seen in point 1. (b) Corresponding image pixels, where clipped pixel is marked by a red circle. C. Sampling with HDR The reconstruction method, explained in the previous sub section assumed that the measured light intensities are linear. Object secularities, and large intensity differences between light sources may introduce non linearities such as sensor noise in dark areas and clipping of values in very bright areas, as seen in figure 3. Removing these non-linearities is important in order to use the reconstruction method explained in the previous section. HDR sampling of still images has been known as a method for enhancing the dynamic range of a camera sensor and has been extensively researched in the past decade [10]. Using HDR for video sequences [11] with Wide Dynamic Range (WDR) sensors [12] is still being investigated. To capture an HDR image, using a standard sensor, one has to take several Low Dynamic Range (LDR) images of a static scene. Each LDR frame has different exposure/gain values of the sensor. The LDR images are then merged using normalization and outlier suppression as explained in [10]. Generating HDR image of changing light intensities, with no control over rate of change of light, with a standard LDR sensors poses a sampling problem (figure 4(a)). This can be solved by capturing the LDR frame in a cyclic manner, as seen in figure 4(b). Real sampling values can be seen in figure 2(a). D. Example application for improving WB and color constancy By using the proposed light source separation it is possible to improve the results of most color balancing and WB algorithms. In this sub-section a simple WB algorithms is

4 (a) Problem with standard sampling (a) Controlled light sources and camera (b) Experiment 1 (b) Proposed sampling method Fig. 4. (a) Example of error that occurs when sampling changing intensities over time with a LDR sensor using 4 Exposure Values (EV). (b) Cyclic sampling which gets an accurate HDR images when light intensity changes over time. improved using light source separation on scene with two light sources. A brief description of the procedure follows: 1) Separate light sources from the mixed light scene 2) Perform WB on each extracted image, which has a single light source. 3) Linearly add all the output images from the previous stage to get the final WB image. IV. A. Prerequisites and equipment EXPERIMENTS To make a proof of concept for this capturing method, a simple experiment was performing using low end camera sensor 1. This subsection may be skipped if a high end camera with frame rate of above 260 FPS is used. Prerequisites: 1) Precise synchronization of camera and light sources was achieved using Matlab software which controlled the light intensity of a led projector and camera properties through a usb connection. Matlab software waited for the camera sensor to finish capturing the frame before preforming the intensity change for the next time lapse. 2) Precise control of light intensity was achieved by setting ten different light intensities, using Matlab software and marked as I n (I 1 = 25, I 2 = 50...I 10 = 250), and measuring them with a light meter 2, marked as L n (L 1, L 2,...L 10 ). Linear function connecting both values such that L n = a In 2 + b and empirically found values for a and b, was estimated. 1 We used the Point Gray c Chameleon T M color camera, 18 FPS 2 Lutron lx-101 Fig. 5. (a) Led projectors used in the first experiment. (b) First experiment diagram where two oscillating lights Lr 1 and Lr 2 were added to S and decomposed using proposed method to get ˆLr 1 and ˆLr 2 3) Precise data capture from sensor was achieved by using the raw data coming from the camera sensor. This helps avoiding gamma correction and tone mapping in the sensor, which usually introduce non linearities to the sensor measurements. B. Experimental setup The experimental setup can be seen in figures 5(a), 6(a) and 6(b). Objects were placed in front of a lambertian white board, which can be seen on the left side of figure 6(b) and right side of figure 6(a). Two experiments were performed: 1) Decomposing two oscillating light sources In this experiment, the white board was illuminated by two oscillating light sources inside a dark room 3. The two light sources were set to have extreme chromatic differences in order to emphasize the efficiency of the proposed separation method, as seen in figure 5. Thirty images were taken while the light sources were modulated by two sinusoidal waveforms. 2) Decomposing an oscillating light from sun light In this experiment several objects were placed in front of a lambertian white board inside a room illuminated by a side window and a single oscillating light source, as seen in figure 6. The single light source was set to have spatial intensity differences across the image, and has a slight chromatic shift to blue. Sun light, coming from the side window, was taken in clear mid day and its intensity stayed constant through the entire video sequence. Thirty images were taken while the light source was modulated by one sinusoidal waveform. V. ANALYSIS AND CONCLUSIONS The experimental results were compared to the reference images. Reference images are images that were captured having only one light source turned on. 3 In this experiment sun light was blocked using curtains

5 sensors will be investigated in future works. REFERENCES (a) Window View (b) Door View Fig. 6. (a) A view from the window side on the experimental setup. Camera and light source can be seen on the left (b) A view from the door side on the experiment setup.camera and light source can be seen on the right (a) Reconstructed light source Vs. real light source (b) Reconstructed sun light Vs. real sun light Fig. 7. (a) Visual comparison of light 1 compared to the ground truth. (b) visual comparison of constant sun light compared to the ground truth. Result and analysis of separating two oscillating light sources Visual comparison of the first experiment can be seen in figure 5(b). Reference images are on the upper part of the diagram, and resulting images are on the lower part of the diagram. Global color of the filtered and reference lights are the same. Difference can be seen mainly in Lr 2 were a purple spot is seen in the center of the filtered light. A careful look at the reference light shows the same spot but with a less saturated color. This means that filtered light has the same pattern as the reference light but is more saturated. This phenomena should be further investigated. Result and analysis of separating oscillating light from sun light Visual comparison of the first experiment can be seen in figures 7(a) and 7(b). Small differences are detected in figure 7(b), were the grey background has a small tint difference (bluish in the reconstructed light and a reddish in the reference image). WB performed on the separated light source and combining them to a single image using simple addition can be visually scene in figures 1(b) and 2(b). A simple max-rgb WB method was used but any other state of the art method may be used instaed. A novel method for separation of oscillating light sources was proposed and demonstrated. The use of high frame rate [1] K. E. Van De Sande, T. Gevers, and C. G. Snoek, Evaluating color descriptors for object and scene recognition, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 32, no. 9, pp , [2] J.-J. Yu, H.-D. Kim, H.-W. Jang, and S.-W. Nam, A hybrid color matching between stereo image sequences, in 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV- CON), IEEE, 2011, pp [3] M. Ebner, Color constancy. John Wiley & Sons, 2007, vol. 6. [4] R. R. Hagege, Scene appearance model based on spatial prediction, Machine Vision and Applications, pp. 1 16, [5] A. Gijsenij, T. Gevers, and J. Van De Weijer, Computational color constancy: Survey and experiments, Image Processing, IEEE Transactions on, vol. 20, no. 9, pp , [6] J.-P. Renno, D. Makris, T. Ellis, and G. A. Jones, Application and evaluation of colour constancy in visual surveillance, in Visual Surveillance and Performance Evaluation of Tracking and Surveillance, nd Joint IEEE International Workshop on. IEEE, 2005, pp [7] N. Wang, B. Funt, C. Lang, and D. Xu, Video-based illumination estimation, in Computational Color Imaging. Springer, 2011, pp [8] V. Prinet, D. Lischinski, and M. Werman, Illuminant chromaticity from image sequences, December [9] D. Poplin, An automatic flicker detection method for embedded camera systems, Consumer Electronics, IEEE Transactions on, vol. 52, no. 2, pp , [10] E. Reinhard, W. Heidrich, P. Debevec, S. Pattanaik, G. Ward, and K. Myszkowski, High dynamic range imaging: acquisition, display, and image-based lighting. Morgan Kaufmann, [11] G. Eilertsen, R. Wanat, R. K. Mantiuk, and J. Unger, Evaluation of tone mapping operators for hdr-video, in Computer Graphics Forum, vol. 32, no. 7. Wiley Online Library, 2013, pp [12] A. Spivak, A. Belenky, A. Fish, and O. Yadid-Pecht, Wide-dynamicrange cmos image sensorscomparative performance analysis, Electron Devices, IEEE Transactions on, vol. 56, no. 11, pp , 2009.