Reconstruction of Images Distorted by Water Waves

Transcription

1 Reconstruction of Images Distorted by Water Waves Arturo Donate and Eraldo Ribeiro Computer Vision Group

2 Outline of the talk Introduction Analysis Background Method Experiments Conclusions Future Work

3 Problem Given a video sequence of an underwater scene, surface waves will cause distortions in the frames. Can we generate a clear image of the underwater scene with minimal geometrical distortion?

4 Analysis Image frame distortion Refraction and blur Non-linear Proportional to size & magnitude of the wave When the water is flat, the image appears completely undistorted Waves can have some flat spots, mainly the crest and trough. By estimating the point in time when the water normal is parallel to the camera s optical axis, we may get a clear view of the underwater scene

5 Refraction Water Surface Underwater Scene p t=0 p t=1 p t=2

6 Previous Methods Averaging (Shefer [7]) simple introduces blurriness Optical flow (Murase [3]) calculates optical flow of pixels estimate surface normals and reconstruct surface handles small distortions optical flow errors lead to inaccurate results

7 Previous Methods Center frame estimation (Efros et al. [2]) Gaussian distribution of frames for one subregion over time Embedding of sub-regions via a weighted graph [2]: Center node: shortest distance to all others

8 Geometric Modeling Refraction follows Snell s Law: Distortions: Linear with respect to d (motion blur) non-linear with respect to α (refraction)

9 Geometric Modeling We know that By substitution, we get: Considering the triangle, the magnitude of the translation is given by: The overall translation is given by:

10 Sample Frames Frames distorted by refraction and blur

11 Distribution

12 Why is averaging undesirable? For a given subregion, amount of distortion applied to each frame varies over time Temporal average combines high and low distortion information For most datasets, there is more high distortion data than low distortion data, resulting in blurry reconstructions

13 Averaging

14 Our Approach Distinguish between clear and distorted frames Handle distortions separately Analyze individual sub-regions Translation analyzed via clustering Motion blur analyzed in the frequency domain

15 Cluster Data Frames are first divided into subregions Each subregion is clustered using K-Means [1] Number of clusters is chosen adaptively

16 Why Clustering? Goal is to group together all minimal-distortion frames Assuming a Gaussian-like distribution, we expect frames closest to the center of the distribution to be similar in appearance and therefore cluster together K-Means has shown good results in clustering these similar frames together

17 Choosing Best Cluster Best cluster is defined as the one containing frames closest in appearance to each other We calculate total variance of pixels to determine which cluster has least amount of relative change

18 Blur Measurement Fourier spectrum of each frame is first computed: A Butterworth high-pass filter is then applied:

19 Why High Frequency? In an image, high-frequency energy content refers to high contrast regions (edges). A blurred edge will have lower highfrequency energy than a sharp edge [5]. Problem: a simple reduction in the number of edges will also cause decrease in high-frequency energy content.

20 High-Frequency Histogram From the filtered power spectrum, we build 1D histograms using [8]: Summing over these histograms provides a way to quantify the amount of blur in each frame.

21 High Energy Comparison Blurred images cause a fast drop of energy in the radial spectral descriptor (bottom row). Measuring this energy allows us to remove the frames containing large amounts of motion blur.

22 Blur Measurement Our method calculates the mean energy of all frames, then removes frames whose high frequency energy is less than the calculated mean. From this final subset we will choose a single frame to represent the given sub-region in the reconstructed image.

23 Distance Matrix We form a distance matrix to find the best frame. Normalized cross correlation is used to compute the distance between all frames. The frame with the shortest overall distance to the others is chosen to represent the sub-region. Similar to Efros et al. [2] transitive distances using NCC leakage problem shortest path algorithm

24 Distance Matrix

25 Algorithm Overview Acquire video Process each set of sub-regions Split into individual frames Divide frames into sub-regions

26 Algorithm Overview Step 1: 4: 2: 3: Cluster Calculate Choose Keep frames sub-regions best distance cluster with most matrix high-frequency to find best single content frame

27 Final Frame Averaging: Our Method:

28 Blending Subregions have 50% overlap with their neighbors. Blending [5] is performed as the last step.

29 Experiments Original video Temporal average Reconstruction

30 Experiments Original video Temporal average Reconstruction

31 Experiments Original video Temporal average Reconstruction

32 Experiments Temporal average Original video Reconstruction

33 Conclusion We propose a new method for reconstructing images of underwater scenes distorted by waves. The novelty of our work comes from the fact that we handle the distortions separately. Experiments show good results in reconstructing scenes under low energy waves. More work is needed to improve reconstruction of scenes under high energy waves. In every experiment, our method achieves better reconstruction results than temporal averaging.

34 Future Work Using Fourier Transform, if the transformation between 2 frames is a translation only (no scale or rotation), then we can estimate the translation parameters via phase correlation. This should provide extra information for choosing the right frames, while also introducing the use of neighboring information.

35 References [1] [5] [2] [3] [4] [6] [7] [8]

36 Translation in Frequency Domain Let f 1 (x) and f 2 (x) be 2 image patches related by an affine transformation only: Their Fourier transforms are related by: If the transformation is a translation only (A = identity matrix), we reduce the previous equation to:

37 Translation in Frequency Domain The translation between f 1 and f 2 corresponds to a shift in frequency domain, determined by the inverse Fourier transform of the phase correlation between the 2 patches [6]:

38 Translation in Frequency Domain The translation is given by the maximum peak of the delta function: