Image Fusion For Context Enhancement and Video Surrealism

Transcription

1 DIP PROJECT REPORT Image Fusion For Context Enhancement and Video Surrealism By Jay Guru Panda ( ) Shashank Sharma( ) Project Idea: Paper Published (same topic) in SIGGRAPH '05 ACM SIGGRAPH 2005 Courses by i) Ramesh Raskar (MERL), Cambridge ii) Adrian Ilie(UNC, Chapel Hill) iii) Jingyi Yu (MIT, Cambridge)

2 Paper Abstract: The primary focus of the paper is on various image fusion techniques to automatically combine images captured under different illumination. Presenting digital tools to create surrealistic images and videos, they apply these methods to practical applications like enhancing the context of night-time traffic videos so that they are easier to understand. Here, the context is automatically captured from a fixed camera and inserted from a day-time image (of the same scene). The approach is based on a gradient domain technique that preserves important local perceptual cues while avoiding traditional problems such as aliasing, ghosting and haloing. Our Work: We have tried to implement the image fusion techniques for images/videos captured in low illumination conditions and enhance them with respect to the context obtained from a day-time(better illumination conditions) image of the same scene. We use the gradient domain approach given in the paper. It was important for us to understand why we choose to process images in the gradient space and why simple spatial domain methods failed. To understand this, we first tried to reason the failure of spatial domain image fusion techniques to merge a night-time and day-time image of the same scene. We tried implementing pure pixel blending strategies and realised the potential problems of ghosting, aliasing and haloing artifacts around the bright areas of night-time images. We then went through the gradient space image processing methodology presented in the paper and understood the technique. The idea is to first encode the pixel importance based on local variance in input images or videos. Then, instead of a convex combination of pixel intensities, use linear combination of the intensity gradients where the weights are scaled by the pixel importance. The image reconstructed from integration of the gradients achieves a smooth blend of the input images, and at the same time preserves their important features.

3 Problem Formulation: Given a night-time image/video as input and a day-time image of the same scene, enhance the night-time image/video to get a surrealistic image/video with its context enhanced. Day-time Image Night-time Image: As can be seen, the Context is hardly identifiable

4 The Traditional Method A naive approach of simple cutting-pasting of pixels from the high-illumination image in its low-illumination counterpart or averaging/maximising will leave problems like ghosting, haloing, etc. We tried implementing spatial domain blending strategies such as max i (Ii (x, y)) or averagei(ii (x, y)). For example, when combining day-night images, one needs to deal with high variance in daytime images and with mostly low contrast and patches of high contrast in night images. Taking the average simply overwhelms the subtle details in the nighttime image, and presents ghosting effects around areas that are bright at nighttime. Furthermore, blending pixels usually leads to visible problems (e.g. sudden jumps from dark night pixels to bright day pixels) that distract from the actual information conveyed in the night images. Context Enhanced night-time image using traditional methods

5 Our Method And Results We aim to capture the information from the night-time image/video and the context from its corresponding day-time image of the same scene. Imagine a black box that does this job for us. Also, we assume another black box which might be the inverse of the first black box that merges these two into the final enhanced image. Night-time image -> Information (I) Day-time image -> Context (C) Final Enhanced image <- I + C The method we followed to achieve may be described step wise as follows: i) Convert both the night-time and day-time spatial image into its gradient domain. ii) To capture the information, we create what is called an Importance Image(W). It is a binary image obtained by thresholding the gradient of night-time image. iii) Then, we combine the gradient of the day-time image with the gradient of night-time image according to the Importance Image(W) to create the Mixed Gradient Image. iv) Finally, we convert the Mixed Gradient Image from gradient domain to spatial domain using the Poisson Solver Algorithm to get the Final Enhanced Image. Importance image of the night-time image that captures the information

6 Mixed Gradient Image X-direction Mixed Gradient Image Y-direction

7 Final Enhanced Image Overall Methodology at a Glance Flow Diagram of the methodology used

8 OTHER EXAMPLES:

9

10

11 FOR VIDEO ENHANCEMENT Additional Challenges: i) inter-frame coherence must also be maintained i.e. Weights in successive frames should change smoothly. ii) a pixel from a low quantity image may be important even if the local variance is smal(e.g., the area between the headlights and the taillights of a moving car). Solution: The idea is again simple. In a sequence of video frames, moving objects span approximately the same pixels from head to tail. For example, the front of a moving car covers all the pixels that will be covered by rest of the car in subsequent frames. Using temporal hysteresis, although the body of a car may not show enough intraframe or inter-frame variance, we maintain the importance weight high in the interval between the head and the tail. The importance is based on the spatial and temporal variation as well as the hysteresis computed at a pixel. A binary mask Mj for each frame Fj is calculated by thresholding the difference with the previous frame, F j - Fj 1. To maintain temporal coherence, we compute the importance image W j by averaging the processed binary masks Mk, for frames in the interval k=j-c..j+c. We chose the extent of influence c, to be 5 frames in each direction. Thus, the weight due to temporal variation W j is a mask with values in [0,1] that vary smoothly in space and time. Then for each pixel of each frame, if Wj (x, y) is non-zero, we use the method of context enhancement of dynamic scene i.e. blend the gradients of the night frame and day frame scaled by Wj and (1 Wj). If Wj (x, y) is zero, we revert to a special case of the method of enhancement for static scenes. Finally, each frame is individually reconstructed from the mixed gradient field for that frame.