Super-resolution on Text Image Sequences

Transcription

1 November 4, 2004

2 Outline Outline Geometric Distortion Optical/Motion Blurring Down-Sampling Total Variation

3 Basic Idea Outline Geometric Distortion Optical/Motion Blurring Down-Sampling No optical/image acquisition system is perfect Types of degradation: Geometric distortion Optical/motion blurring Down-sampling Additive noise Our goal is to compensate for this and produce a higher resolution image given a sequence of low-resolution images.

4 Geometric Distortion Outline Geometric Distortion Optical/Motion Blurring Down-Sampling image plane planar surface optic centre quantization

5 Optical/Motion Blurring Outline Geometric Distortion Optical/Motion Blurring Down-Sampling Optical blurring in eye: 1 1

6 Optical/Motion Blurring Outline Geometric Distortion Optical/Motion Blurring Down-Sampling Figure 5.5: Each simulated pixel is a weighted sum of several super-pixels. The weights are determined by the geometric viewing transformation, the form and size of the pointspread function, and also by the quadrature rule used to discretize the generative model. PSF is usually unknown, but Capel and Zisserman chose Gaussian and saw good results. Essentially, there are three possible approaches to the discretization of the generative model which we shall now examine.

7 Down-Sampling Outline Geometric Distortion Optical/Motion Blurring Down-Sampling image plane planar surface optic centre quantization Figure 5.3: (Top) A perspective camera is imaging a planar scene. (Left) The scene is projected onto the CCD array by perspective projection. (Right) The CCD elements are assumed to integrate the radiant energy over their surface, producing a spatially-quantized image of the scene.

8 Model Hi-Res Image Filter Low-Res Image Need Mathematical model that represents process At each point (x, y): m n = S (h T [l(s)]) + η (1) m n (x, y) = S (h(u, v) s (T (x, y))) + η (2)

9 In Matrix Form... Model Becomes: m n = S (h T [l(s)]) + η m n = M n s + η (3) where h, T, S are combined in the matrix M n. For all images, we can stack vertically for an over-determined linear system: m 0 m 1. m N 1 = M 0 M 1. M N 1 + η 0 η 1. η N 1 (4) m = Ms + η (5)

10 Estimator Total probability of observed image m n, given estimate of super-resolution image ŝ is: Pr(m n ŝ) = 1 σ 2π e (bmn(x,y) mn(x,y)) x,y And the associated log-likelihood function is: 2 2σ 2 (6) L(m n ) = x,y ( m n (x, y) m n (x, y)) 2 (7)

11 Estimator To find the maximum likelihood estimate, s ML, we need to maximize L(m n ) over all images: s ML = argmax s L(m n ) (8) Using the Matrix form for the image model from before, we have: L(m n ) = n n n M n s m n 2 = Ms m 2 The maximization above, becomes equivalent to: s ML = argmin s Ms m 2

12 Estimator This is now a standard minimization problem: s ML = (M T M) 1 M T m (9) }{{} M+ Because M is very large and sparse and Nn 2 m 2 matrix and typically N = 30, n 2 = 2500, m 2 = 10000, it is impractical to compute pseudo-inverse. Need iterative methods like conjugate gradient.

13 Problems Very sensitive to noise:

14 Bounded To Solve, use Bounded Estimator:

15 Minimizes same cost function as ML estimator, but uses iterative update with error back-projection: s i+1 = s i + 1 C n T 1 n [h bpf S ( m n m n )] (10) where the back-projection function h bpf = (h psf ) k, for some k 1.

16 Results

17 Problems Very slow to converge compared to Conjugate-Gradient minimization method in MLE:

18 In the MLE, we used: Pr(m n ŝ) = 1 σ 2π e (bmn(x,y) mn(x,y)) 2σ 2 x,y Assuming independent observations, the total probability over all images in sequence is: 2 Pr(m ŝ) = n Pr(m n ŝ) (11) Using Bayes s theorem: Pr(ŝ m) = Pr(m ŝ)pr(ŝ) Pr(m) (12)

19 The maximum a-posterior (MAP) estimate of s becomes: s MAP = argmax s = argmax s Taking the logarithms, we get: Pr(m ŝ)pr(ŝ) Pr(m) Pr(m ŝ)pr(ŝ) s MAP = argmax s = argmax s lnpr(m ŝ) + lnpr(ŝ)) (13) L(m n ) + L(s) (14) n

20 Likelihood of s estimate How do we know what L(s) is? In this paper, Capel and Zisserman suggest a Huber cost function, f (x): s MAP = argmax s L(m n ) λ 2 f ( s(x, y)) (15) n x,y f (x) = x 2, if x α = 2α x α 2, otherwise

21 Likelihood of s estimate But, why the Huber function? Allows for local smoothness while being lenient toward step edges

22 Results

23 Same idea as MAP estimator, but use different prior: s TV = argmax L(m n ) λ 2 s(x, y) (16) s n x,y Since, dtv ds =. s s, we can use s s 2 x + s 2 y + β to avoid singularity at s = 0.

24 Mosaicking

25 License Plate Recognition

26 How do we get from Eqn 1 to Eqn 2? Are we using a common region of the low resolution frames to get the super-resolution image? Otherwise, some frames may not contain the whole area of the image that we are interested in.

27 Recommended Resources David Capel s PhD Thesis: Image Mosaicing and Super-resolution, 2001 Capel and Zisserman. Computer Vision Applied to Super-resolution. IEEE Signal Processing Magazine (2003) vgg/publications/papers/capel03.pdf