A Study on Blur Kernel Estimation from Blurred and Noisy Image Pairs

Transcription

1 A Study on Blur Kernel Estimation from Blurred and Noisy Image Pairs Mushfiqur Rouf Department of Computer Science University of British Columbia Abstract The course can be split in two parts. In the implementation part, the kernel estimation process as described in [1] has been studied. The algorithm has been tested against synthetic and real data; and its performance has been discussed. In the reading part, a number of papers have been covered from the list of papers discussed in the graduate level course CPSC 505 Image Understanding I: Image Analysis in Winter Term I, Introduction Image deconvolution is a widely studied yet largely unsolved problem. This study has been carried out on Landweber s algorithm, an iterative variation of the Tikhonov Regularization technique as is described in [1]. An in-depth analysis has been performed to understand the behavior of Landweber s algorithm, which is an iterative method, when performed with Hysteresis thresholding A Walk through the Algorithm The algorithm in [1] can be described to have two steps: 1. Kernel Estimation using Landweber method with Hysteresis Thresholding; 2. Deconvolution using gain controlled Richardson Lucy followed by a detail adding step. In this study, the first step has been investigated. This study started with a simple implementation of the algorithm as described in the paper, and then the implementation was modified in order to study its behavior under different circumstances Landweber Method Landweber method invokes the Tikhonov regularization algorithm [2] in an iterative fashion. Convolution operation can be described as a vector multiplication by the blur kernel k, Ak = b Here, A is matrix where each row is the neighborhood of a pixel in the original image and b is a column vector containing the corresponding pixels from the blurred image. In the implementation, pixel values lie between 0 and 1 This is a CPSC 548 Direct Graduate Studies course report. This study has been carried out under the supervision of Dr. Wolfgang Heidrich. inclusive; and 8-bit depth has been assumed. The iterative Tikhonov regularization can be expressed as an iterative matrix inversion algorithm that starts with an initial guess and converges to k = A -1 b: k k ( 0) = δ ( n+ 1) ( n) T T 2 ( n) = k + β ( A b ( A A + λ I) k ) Here, δ is the Dirac Delta function, β is the convergence parameter with a default value of 1 and λ is the regularization parameter with a default value of 5 [1]. However, since A is unknown, the noisy image was used in place of A. It has been shown in [1] that the noisy image can be a reasonable estimate to compute k effectively Hysteresis Thresholding Regularization is done at different scale levels. Original image is scaled down by a factor of 1/ 2 in each scale level, stopping at the size of 9x9. Thresholded versions of predicted lower resolution kernels are used as masks for the upper levels in order to reduce noise. In the implementation, cubic interpolation was used to scale down the image for hysteresis thresholding Contribution 1. This report gives a study of parameters described in the kernel estimation portion of [1]. 2. This report investigates the effectiveness of the Landweber method and presents the weaknesses of this algorithm. 3. This report gives the results of running the algorithm on real datasets and describes a possible reason why it fails Organization of the Report The remaining of the report are organized as follows. In section 2, the experiments and observations are described. Results are summarized and the effectiveness of the Kernel estimation as described in [1] is evaluated in section 3. Section 4 gives the conclusion of this report. Section 4 also points to some possible future research directions. Section 7 gives a list of papers read for the reading part. 2. Observations The kernel estimation algorithm has been implemented in Matlab and tested with both synthetic and real datasets.

2 (a) β set to 1 and remains unchanged. (a) β set to and remains unchanged. (c) β initialized to and reduced by 5% when (d) β initialized to and reduced by 5% when necessary necessary Figure 1: The oscillation behavior. (a) and (b) shows the oscillation occurred even for a small value for β. (c) shows convergence can be achieved with a varying β. In this study, the favorable initial value of β has been found to be , which resulted in a faster convergence (d). Image 5 (Figure 4(c)) and kernel 6 (Figure 4(a)) were used to produce the above results. Synthetic data have been created using a number of test kernels, applied over a set of images collected from the internet. All computations are done on grayscale versions Initial Implementation The initial implementation uses default parameter values as mentioned in [1]. The investigation started with synthetic data; real data was input to the system once the system was stabilized for all synthetic cases Speed up It is found that the most time consuming step is computing A T A. A speed up was designed using the fact that most of A is redundant since the rows represent neighborhood of adjacent pixels Oscillatory Behavior The initial implementation failed to predict the kernel in all cases. The Tikhonov iterations seem to toggle between two different states (Figure 1), none of them being the solution. In most cases the states were vertically, horizontally or diagonally opposite mirrored with respect to one another. In some cases, a diverging oscillation seemed to build up gradually; at each iteration, picking a state further from the solution than the previous state. In some cases, the algorithm would come very close to a stable solution and then drift away to oscillation. These phenomena strongly suggested the algorithm is trying to converge to a solution but is being prevented by some wrong choice of parameter value. To stabilize the algorithm, the parameters were needed to be tuned. λ was tried with first, then β Role of λ and the Condition Number Blindly trying with different λ values ranging from 0 to 100 did not yield any better result. The Landweber method is essentially a matrix inversion algorithm; therefore a high condition number reduces the chance of getting a meaningful result. Initially the reason behind the oscillatory behavior was suspected to be poor conditioning of A. For a number of datasets, it was found that λ should have a value in the range of thousands to get the condition number down to the range of thousands. Unfortunately, a high value of λ also means a heavy diagonal was being added, essentially rendering the information in A useless. It was understood that lower λ values would result in a low condition number which would introduce computational errors, but these errors were not responsible for the oscillations that were observed Keeping the negative values in k A blur kernel must have nonnegative values at each pixel and the sum of all values must be equal to one. However, the Tikhonov regularization iterations were producing negative values; and the initial solution was to put a zero at the negative value positions; thus clipping the estimated kernel to a zero. However, it was observed that the discarded negative values were becoming positive in the next iteration. This might have caused the oscillations, so the negative values were being kept for first few iterations; and the clipping off at zero began from the fifth iteration. The motivation of doing this was that the initial guess (Delta function) was a very far state from the actual solution and losing information at such early steps may lead to a wrong solution. After first few steps the state became sufficiently close and clipping resumed.

3 2.5. Finding the right β The oscillation seemed to be happening around the solution. That means the convergence rate needed to be slowed down so that the algorithm could converge. By lowering down the value of β the algorithm could be made to converge to a solution. But low β means slow convergence. Also, it was found that the appropriate value of β was dependent on the image and the blur kernel. Consequently, finding an optimum β was needed to be done automatically during run time Finding optimum β The algorithm started with β = Oscillations can be detected by analyzing three consecutive kernel estimations. If dist(k n, k n-1 ) > dist(k n, k n-2 ) then there is an oscillation. 2-norm was used as the distance metric. In case of an oscillation, the value of β was reduced by 5%. That would slow down the process and allow the algorithm to converge at a solution. Similarly, right at the beginning, β was increased for a faster convergence. This way the algorithm could start with a high β to execute a rapid search in the search space and slow down as it got near the solution (Figure 1). satisfactory result in all cases. In most cases the stochastic jump reached the same solution Effect of edges Interestingly, strong edges in the image have strong effect on the kernel estimation. The kernel estimations consistently show distortions parallel to the strong edges in the original image (Figure 4). This biasness can be explained by the fact that having edges means having a set of points with the same neighborhood. The Landweber method is a least square optimization approach; having a set of pixels with the same neighborhood means an aspect getting more weight towards minimization. That way the least squared error can be reduced greatly if these clusters of pixels are satisfied Levin Filter The Levin filter was tried on the synthetic data. Different sizes and rotations of the filter have been tried with. The algorithm successfully produced a good estimate of the kernel (Figure 3). Different sizes and orientations are generated using cubic interpolation Convergence In all the experiments, the kernel estimation converged to a solution. There is no mathematical proof that the algorithm will converge at the global optimum, so a stochastic search was implemented on top of the Tikhonov iteration Introducing Randomness After the Tikhonov iteration converged, a white noise was added to the predicted kernel to run Landweber method again (Figure 2), with this kernel as the initial guess, instead of the Dirac Delta function. The experiments show that estimated kernel is indeed improved by a small proportion in a small number of cases, and four such stochastic jumps are found to produce (a) Tikhonov iteration (c) (d) (b) Stable iterations (e) Figure 2: White noise was added 3 times after reaching convergence each time. The algorithm still converges to the same solution. Figure 3: Synthetic application of the Levin filter. (a) shows the states in a series of iterations. (b) shows stable condition after applying the stochastic jump three times. (c) shows the actual Levin filter, (d) shows the rotated version used to blur the image, and (e) shows the estimated PSF. Test image 1 (Figure 4(c)) was used.

4 (a) 14 blur kernels that were used to synthetically blur the test images. (b) i-th kernel on each block is predicted from a noisy and (c) Test images. blurred image pair produced by applying i-th kernel from the top left block to the corresponding image on the right. Figure 4: The effect of edges on the deconvolution algorithm. For better viewing, small kernels are scaled up and kernel pixel values are scaled such that for each kernel the highest pixel value maps to 1. (a) (b) (c) Figure 5: Real data analysis. (a) and (b) show portions of the blurred and noisy image pair. The non-smooth artifacts in (a) confirm that the blur kernel is not Gaussian. However, the algorithm generated a Guassian PSF estimate (c) Real Data Real data have been collected from the HDR Defocus project at PSM (path to the data files: /ubc/cs/research/psmraid1/hdrdefocus/ /secondscene/raw/). These data had been originally collected using the Levin filter. For this study, 400x400 pixel crops from the center of the images were used. For the real data, the algorithm miserably failed to estimate the kernel (Figure 5). In all cases the algorithm produced only Gaussians kernel. The algorithm could find the size and a rough shape of the kernel. When the Levin filter was slightly stretched and rotated, the predicted Gaussian had the same transformation as well. The question was why the real data failed although the synthetic data did not fail. To understand why real data failed, different circumstances were to be synthesized so that the synthetic case would fail. A number of approaches are taken; the results obtained are discussed here Adding Noise At first, 1% noise was added to the synthetically blurred image to see if the algorithm now failed. But the algorithm continued to find a reasonable kernel estimate.

5 Quantization At this point, it has been found that the implementation was not quantizing the blurred image after computing the convolution. This way the convoluted image was conveying reasonable amount of information that the real data could not, due to precision limit. However, even after quantizing the blurred image to 256 steps (8 bits of information per pixel), the implementation continued to converge at similar estimates of the kernels as before Adding a Fixed Offset When the real data were observed, a small offset was found in the min, max and average pixel values. When such a small 0.1% offset was added to the blurred pixel values, the algorithm failed to predict a reasonable kernel, instead it produced only a Gaussian. In the real data the small offset was not the same in min, average and max pixel values. Apparently the fixed offset was accompanied by a variable scaling factor at different pixel values and locations. This may happen due to the changing black level of the image taken by a camera. The black level, in turn, depends on the temperature of the camera. 3. Results The algorithm works fine with synthetic data; it can make a good estimate of any kernel. This algorithm has also been found to withstand a noise of 1%. However, in real data, it fails; which is found to occur due to inconsistencies in corresponding pixel values. 4. Conclusion and Future Work The algorithm has a tendency of producing Gaussian kernel estimations. It may be difficult to control the black level of a camera while taking the blurred and noisy pairs of the image since the blurred image is taken using a long exposure and the noisy image is taken using a short exposure. Another iterative approach may be developed. The idea is to apply certain preprocessing on the blurred and noisy images so that the offset and the scaling effects of the black level can be reduced. In addition, if the filter is known, a first Kernel estimation can give a reasonable estimation of the size and shape of the kernel. This information may be used to reduce these effects to run the whole kernel estimation algorithm again, now producing a better result. 5. Acknowledgement I must thank Dr. Wolfgang Heidrich for giving me the opportunity to explore this challenging and interesting research area. My special thanks to Matthew Trentacoste for the initial Landweber code he provided, for giving a brief overview of how to program using Matlab and for coming up with solutions to my problems related to the study at a number of occasions. 6. References [1] L. Yuan, J. Sun, L. Quan and H.-Y. Shum, Image Deblurring with Blurred/Noisy Image Pairs, ACM SIGGRAPH, [2] A. Neumaier, Solving ill-conditioned and singular linear systems: A tutorial on regularization, SIAM Review 40, pp , List of Papers Read The papers starting at [2] are selected from the list of papers covered in the graduate level course CPSC 505 Image Understanding I: Image Analysis in Winter Term I, A complete list of papers can be found here: ml. Papers have been read from sampling theory, edge detection, optical flow and visual tracking. [1] Q. Shan, W Xiong and J. Jia, Rotational Motion Deblurring of a Rigid Object from a Single Image, in Proceedings of IEEE 11 th Conference on Computer Vision, pp 1-8, [2] M. Unser, Sampling - 50 years after Shannon, Proceedings of the IEEE, vol. 88, no. 4, pp , [3] C. E. Shannon, Communication in the presence of noise, Proc. IRE, vol. 37, pp , [4] D. Marr and E. Hildreth, Theory of edge detection, Proc. R. Soc. Lond. B, vol. 207, pp , [5] J. F. Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp , [6] R. Deriche, Using Canny's criteria to derive a recursively implemented optimal edge detector, International Journal of Computer Vision, vol. 1, pp , [7] V. Torre and T. A. Poggio, On edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 8, no. 2, pp , [8] L. Zhang, B. Curless, A. Hertzmann, and S. M. Seitz, Shape and motion under varying illumination: unifying structure from motion, photometric stereo, and multi-view stereo, in Proc. 9th International Conference on Computer Vision, pp , [9] J. L. Barron, D. J. Fleet, and S. S. Beauchemin, Performance of optical flow techniques, International Journal of Computer Vision, vol. 12, no. 1, pp , 1994.

6 [10] R. J. Woodham, Multiple light source optical flow, in Proc. 3rd International Conference on Computer Vision, (Osaka, Japan), pp , [11] R. J. Woodham, Multiple light source optical flow, in Proc. 3rd International Conference on Computer Vision, (Osaka, Japan), pp , [12] C. Stauffer and E. Grimson, Learning patterns of activity using real-time tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp , [13] A. D. Jepson, D. J. Fleet, and T. F. El-Maraghi, Robust online appearance models for visual tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp , [14] D. G. Lowe, Distinctive image features from scaleinvariant keypoints, International Journal of Computer Vision, vol. 60, no. 2, pp , [15] K. Okuma, A. Taleghani, N. de Freitas, J. J. Little, and D. G. Lowe, A boosted particle filter: Multitarget detection and tracking, in Proc. 8th European Conference on Computer Vision, vol. 1, pp , 2004.