Introduction to Image Super-resolution. Presenter: Kevin Su

Transcription

1 Introduction to Image Super-resolution Presenter: Kevin Su

2 References 1. S.C. Park, M.K. Park, and M.G. KANG, Super-Resolution Image Reconstruction: A Technical Overview, IEEE Signal Processing Magazine, Vol. 20, pp , May W.T. Freeman, T.R. Jones, and E.C. Pasztor, Example-Based Super-Resolution, IEEE Computer Graphics and Applications, Vol. 22, pp , 2002.

3 Overview Introduction Super-resolution Techniques Multi-frame Super-resolution Single-frame Super-resolution Conclusion

4 Terminology Low-Resolution (LR): Pixel density within an image is small, therefore offering less details. High-Resolution (HR): Pixel density within an image is larger, therefore offering more details. Superresolution (SR): Obtaining a HR image from one or multiple LR images.

5 Application Medical imaging (ie. CAT, MRI, etc). Satellite imaging Enlarging consumer photographs Video surveillance (ie. Car wash kidnapping). Converting NTSC video content to highdefinition television

6 Application raw After apply super-resolution technique

7 Application zoom apply super-resolution technique

8 Application Video with low resolution Video with high resolution

9 Application

10 How to increase resolution? Possible ways for increasing an image resolution: Reducing pixel size. Increase the chip-size. Super-resolution.

11 How to increase resolution? Reduce pixel size: Increase the number of pixels per unit area. Advantage: Increases spatial resolution. Disadvantage: Noise introduced. As the pixel size decreases, the amount of light decreases.

12 How to increase resolution? Increase the chip size (HW): Advantage: Enhances spatial resolution. Disadvantage: High cost for high precision optics.

13 How to increase resolution? Superresolution (SR): Process of combining multiple low resolution images to form a high resolution image. Advantages: Cost less than comparable approaches. LR imaging systems can still be utilized.

14 Overview Introduction Super-resolution Techniques Multi-frame Super-resolution Single-frame Super-resolution Conclusion

15 Multi-frame Super-resolution How can we obtain a HR image from multiple LR images? Basic premise is the availability of multiple LR image captured form the same scene. These multiple LR images provide different looks at the same scene. Each LR is naturally shifted with subpixel precision. If LR images are shifted by integer units, then each image contains the same information, SR is not possible. If LR images have different subpixel shifts, then SR is possible.

16 Basic premise for SR

17 Common image acquisition System

18 Observation Model First step to understanding SR is to formulate an Observation Model to relate the LR images to the desired HR image.

19 Desired HR image: Size: L 1N1 L2N 2 T Vector: where LR images: Observation Model x = x, x,..., ] [ 1 2 Size: N 1 N 2 K-th LR image: where =,, x N N = L N y k = [y ] k,1,yk,2,...,yk,m k , p and M = N N L2N 2 T

20 Observation Model Observation model can be represented as follows: y k = DB M x + n for 1 k M k is a warp matrix B k represents blur matrix D is a sub-sampling matrix represents noise matrix n k k Without loss of generality, it can also be represented as follows: y k = W x + n for 1 k k k k k p p

21 Nonuniform interpolation 3 stages: Registration Interpolation Deblurring approach

22 Nonuniform interpolation Registration: approach Need to estimate the scene motion for each image with reference to one particular image. The motion can be estimated as a 1-to-1 representation between the reference image and each of the images.

23 Registration: Nonuniform interpolation approach Estimating the completely arbitrary motion in real world image scenes is extremely difficult, with almost no guarantees of estimator performance. Incorrect estimates of motion have disastrous implications on overall SR performance.

24 Nonuniform interpolation Interpolation: approach Since the shifts between the LR images are arbitrary, the images will not always match up to a uniformly to the HR grid. Thus, nonuniform interpolation is necessary to obtain a uniformly spaced HR image from a nonuniformly spaced composite of LR images. Nonuniform interpolation between LR images are used to improve resolution.

25 Nonuniform interpolation approach Interpolation: This step requires interpolation when the estimated fractional unit of motion is not equal to the HR grid reference image.

26 Nonuniform interpolation approach Deblurring: In SR, blur is usually modeled as a spatial averaging operator as shown below.

27 Result

28 Regularized SR Reconstruction If there are enough LR images, we can solve y k = W x + n for 1 In reality, it is hard to find sufficient number of LR images. Use procedure (called regularization) to stabilize the inversion of illposed problem. Deterministic Approach (CLS) Stochastic Approach (MAP) k k k p

29 Deterministic Approach (CLS) CLS can be formulated by choosing x to minimize the Lagrangian p k = 1 y k W x k 2 + α Cx C is generally a high-pass filter α is regularization parameter The cost function above is convex and differentiable with the use of a quadratic regularization term. We can find a unique estimate image using iterative techniques xˆ p p T T Wk Wk + αc C xˆ = k = 1 k = 1 W T k y k

30 Stochastic Approach (MAP) Bayesian approach provides a flexible and convenient way to model a priori knowledge concerning solution x = arg max P( x y1, y2,..., y p ). x = arg max{ln P( y1, y2,..., y p x) + ln P( x)}. Using MRF Gibbs priori to define P(x) 1 1 P( X = x) = exp{ U ( x)} = exp( ϕc( x)) Z Z c S

31 Result

32 Other Approaches Frequency Domain Approach Projection onto Convex Sets Approach ML (maximum likelihood approach) ML-POCS hybrid approach Iterative back-projection approach Adaptive Filtering Approach Motionless SR Reconstruction Approach

33 Overview Introduction Super-resolution Techniques Multi-frame Super-resolution Single-frame Super-resolution Conclusion

34 Overview Introduction Super-resolution Techniques Multi-frame Super-resolution Single-frame Super-resolution Conclusion

35 Single-frame SR Traditional resolution enhancement: Smoothing (Gaussian, Wiener, and median filters) Interpolation (Nearest ngbr, bilinear, bicubic and cubic spline etc) Sharpening by amplifying existing image details (it is useful to do, provided noise isn t amplified) Single-frame SR: Estimate missing high-resolution detail that isn t present in the original image, and which we can t make visible by simple sharpening

36 Example-based SR Algorithm uses a training set to learn the fine details of an image at lowresolution. It then uses those learned relationships to predict fine details in other images. Markov network One pass algorithm

37 Training Set Generation Start with a collection of HR images. For each HR image, degrade it to get a LR image. Blur & subsample each to create LR image of ¼ total pixels. Apply analytical interpolation to the LR image. ie. Cubic spline. This will generate an image of desired # of pixels, but lacking the HR detail. Band pass filter and contrast normalize the interpolated image AND the original HR image.

38 Training Set Generation

39 Training Set Generation Divide images into small patches: 5x5 (HR), 7x7 (LR)

40 Markov network

41 Markov network Select the 16 or so closet examples to each input patch as the different states of the hidden nodes, x, that we seek to estimate. Maximize 1 P( x y) = ψ (, ) ij xi x j φk ( xk, yk ) Z where d ψ ( x ij i, x ( x, x j ) j ) ( ij) d exp ij i = 2 2σ ij i, the sum of squared differences between patch candidates x i and x j in their overlap regions at nodes i and j ( x, x j ) k

42 One pass algorithm

43 Results

44 Training Set

45 Results

46 Conclusions and future works Current SR approaches are effective to some extent SR considering registration error: Use total least squares method to minimize the error Use channel adaptive regularization: SR images with large registration error should be less contributed to the estimate of the HR. Blind SR Image Reconstruction: when blurring process is unknown. Need blur identification. Computationally efficient SR Algorithm