Introduction to Image Super-resolution Presenter: Kevin Su
References 1. S.C. Park, M.K. Park, and M.G. KANG, Super-Resolution Image Reconstruction: A Technical Overview, IEEE Signal Processing Magazine, Vol. 20, pp. 21-36, May 2003 2. W.T. Freeman, T.R. Jones, and E.C. Pasztor, Example-Based Super-Resolution, IEEE Computer Graphics and Applications, Vol. 22, pp. 56-65, 2002.
Overview Introduction Super-resolution Techniques Multi-frame Super-resolution Single-frame Super-resolution Conclusion
Terminology Low-Resolution (LR): Pixel density within an image is small, therefore offering less details. High-Resolution (HR): Pixel density within an image is larger, therefore offering more details. Superresolution (SR): Obtaining a HR image from one or multiple LR images.
Application Medical imaging (ie. CAT, MRI, etc). Satellite imaging Enlarging consumer photographs Video surveillance (ie. Car wash kidnapping). Converting NTSC video content to highdefinition television
Application raw After apply super-resolution technique
Application zoom apply super-resolution technique
Application Video with low resolution Video with high resolution
Application
How to increase resolution? Possible ways for increasing an image resolution: Reducing pixel size. Increase the chip-size. Super-resolution.
How to increase resolution? Reduce pixel size: Increase the number of pixels per unit area. Advantage: Increases spatial resolution. Disadvantage: Noise introduced. As the pixel size decreases, the amount of light decreases.
How to increase resolution? Increase the chip size (HW): Advantage: Enhances spatial resolution. Disadvantage: High cost for high precision optics.
How to increase resolution? Superresolution (SR): Process of combining multiple low resolution images to form a high resolution image. Advantages: Cost less than comparable approaches. LR imaging systems can still be utilized.
Overview Introduction Super-resolution Techniques Multi-frame Super-resolution Single-frame Super-resolution Conclusion
Multi-frame Super-resolution How can we obtain a HR image from multiple LR images? Basic premise is the availability of multiple LR image captured form the same scene. These multiple LR images provide different looks at the same scene. Each LR is naturally shifted with subpixel precision. If LR images are shifted by integer units, then each image contains the same information, SR is not possible. If LR images have different subpixel shifts, then SR is possible.
Basic premise for SR
Common image acquisition System
Observation Model First step to understanding SR is to formulate an Observation Model to relate the LR images to the desired HR image.
Desired HR image: Size: L 1N1 L2N 2 T Vector: where LR images: Observation Model x = x, x,..., ] [ 1 2 Size: N 1 N 2 K-th LR image: where =,, x N N = L N y k = [y ] k,1,yk,2,...,yk,m k 1 2..., p and M = N N 1 2 1 1 L2N 2 T
Observation Model Observation model can be represented as follows: y k = DB M x + n for 1 k M k is a warp matrix B k represents blur matrix D is a sub-sampling matrix represents noise matrix n k k Without loss of generality, it can also be represented as follows: y k = W x + n for 1 k k k k k p p
Nonuniform interpolation 3 stages: Registration Interpolation Deblurring approach
Nonuniform interpolation Registration: approach Need to estimate the scene motion for each image with reference to one particular image. The motion can be estimated as a 1-to-1 representation between the reference image and each of the images.
Registration: Nonuniform interpolation approach Estimating the completely arbitrary motion in real world image scenes is extremely difficult, with almost no guarantees of estimator performance. Incorrect estimates of motion have disastrous implications on overall SR performance.
Nonuniform interpolation Interpolation: approach Since the shifts between the LR images are arbitrary, the images will not always match up to a uniformly to the HR grid. Thus, nonuniform interpolation is necessary to obtain a uniformly spaced HR image from a nonuniformly spaced composite of LR images. Nonuniform interpolation between LR images are used to improve resolution.
Nonuniform interpolation approach Interpolation: This step requires interpolation when the estimated fractional unit of motion is not equal to the HR grid reference image.
Nonuniform interpolation approach Deblurring: In SR, blur is usually modeled as a spatial averaging operator as shown below.
Result
Regularized SR Reconstruction If there are enough LR images, we can solve y k = W x + n for 1 In reality, it is hard to find sufficient number of LR images. Use procedure (called regularization) to stabilize the inversion of illposed problem. Deterministic Approach (CLS) Stochastic Approach (MAP) k k k p
Deterministic Approach (CLS) CLS can be formulated by choosing x to minimize the Lagrangian p k = 1 y k W x k 2 + α Cx C is generally a high-pass filter α is regularization parameter The cost function above is convex and differentiable with the use of a quadratic regularization term. We can find a unique estimate image using iterative techniques xˆ p p T T Wk Wk + αc C xˆ = k = 1 k = 1 W T k y k
Stochastic Approach (MAP) Bayesian approach provides a flexible and convenient way to model a priori knowledge concerning solution x = arg max P( x y1, y2,..., y p ). x = arg max{ln P( y1, y2,..., y p x) + ln P( x)}. Using MRF Gibbs priori to define P(x) 1 1 P( X = x) = exp{ U ( x)} = exp( ϕc( x)) Z Z c S
Result
Other Approaches Frequency Domain Approach Projection onto Convex Sets Approach ML (maximum likelihood approach) ML-POCS hybrid approach Iterative back-projection approach Adaptive Filtering Approach Motionless SR Reconstruction Approach
Overview Introduction Super-resolution Techniques Multi-frame Super-resolution Single-frame Super-resolution Conclusion
Overview Introduction Super-resolution Techniques Multi-frame Super-resolution Single-frame Super-resolution Conclusion
Single-frame SR Traditional resolution enhancement: Smoothing (Gaussian, Wiener, and median filters) Interpolation (Nearest ngbr, bilinear, bicubic and cubic spline etc) Sharpening by amplifying existing image details (it is useful to do, provided noise isn t amplified) Single-frame SR: Estimate missing high-resolution detail that isn t present in the original image, and which we can t make visible by simple sharpening
Example-based SR Algorithm uses a training set to learn the fine details of an image at lowresolution. It then uses those learned relationships to predict fine details in other images. Markov network One pass algorithm
Training Set Generation Start with a collection of HR images. For each HR image, degrade it to get a LR image. Blur & subsample each to create LR image of ¼ total pixels. Apply analytical interpolation to the LR image. ie. Cubic spline. This will generate an image of desired # of pixels, but lacking the HR detail. Band pass filter and contrast normalize the interpolated image AND the original HR image.
Training Set Generation
Training Set Generation Divide images into small patches: 5x5 (HR), 7x7 (LR)
Markov network
Markov network Select the 16 or so closet examples to each input patch as the different states of the hidden nodes, x, that we seek to estimate. Maximize 1 P( x y) = ψ (, ) ij xi x j φk ( xk, yk ) Z where d ψ ( x ij i, x ( x, x j ) j ) ( ij) d exp ij i = 2 2σ ij i, the sum of squared differences between patch candidates x i and x j in their overlap regions at nodes i and j ( x, x j ) k
One pass algorithm
Results
Training Set
Results
Conclusions and future works Current SR approaches are effective to some extent SR considering registration error: Use total least squares method to minimize the error Use channel adaptive regularization: SR images with large registration error should be less contributed to the estimate of the HR. Blind SR Image Reconstruction: when blurring process is unknown. Need blur identification. Computationally efficient SR Algorithm