Super-Resolution Many slides from Mii Elad Technion Yosi Rubner RTC and more 1
Example - Video 53 images, ratio 1:4 2
Example Surveillance 40 images ratio 1:4 3
Example Enhance Mosaics 4
5
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Special cases and related problems SR in time SR from Examples 6
Intuition For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough ( D), perfect reconstruction is possible. D D 7
Intuition Due to our limited camera resolution, we sample using an insufficient 2D grid 2D 2D 8
Intuition However, if we tae a second picture, shifting the camera slightly to the right we obtain: 2D 2D 9
Intuition Similarly, by shifting down we get a third image: 2D 2D 10
Intuition And finally, by shifting down and to the right we get the fourth image: 2D 2D 11
Intuition It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed. 12
Rotation/Scale/Disp. What if the camera displacement is Arbitrary? What if the camera rotates? Gets closer to the object (zoom)? 13
Rotation/Scale/Disp. There is no sampling theorem covering this case 14
A Small Example 3:1 scale-up in each axis using 9 images, with pure global translation between them 15
Further Complications Complicated motion perspective, local motion, Blur sampling is not a point operation Spatially variant blur Temporally variant blur Noise Changes in the scene 16
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Special cases and related problems SR in time SR from Examples 17
Image Formation Scene HR Geometric transformation F Optical Blur H Sampling D Image LR Can we write these steps as linear operators? LR H F HR = D 18
Geometric Transformation F Scene Geometric transformation Any appropriate motion model Every frame has different transformation Usually found by a separate registration algorithm 19
Geometric Transformation Can be modeled as a linear operation F X F = X F X F X 20
Optical Blur H Geometric transformation Optical Blur Due to the lens PSF and pixel integration Usually H = H 21
H PSF * PIXEL = H 22
Optical Blur Can be modeled as a linear operation HX H = X HX H X 23
Sampling D Optical Blur Sampling Pixel operation consists of area integration followed by decimation D is the decimation only Usually D = D 24
Decimation Can be modeled as a linear operation DX D X DX o 1 0 1 0 o 1 0... 1 0 D 1 0 X = 25
Image Formation Scene HR Geometric transformation F Optical Blur H Sampling D Image LR LR= D H F HR 26
Image Formation + Scene HR Image LR Noise N LR D H F HR+ = N 27
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Special cases and related problems SR in time SR from Examples 28
Super-Resolution - Model Geometric warp Blur Decimation High- Resolution Image X F =I 1 H 1 D 1 V 1 Y 1 Low- Resolution Exposures Additive Noise F N H N D N Y N Y V N { 2 0, } = D H F X + V, V ~ N σ n N = 1 29
Simplified Model Geometric warp Blur Decimation High- Resolution Image X F =I 1 H D V 1 Y 1 Low- Resolution Exposures Additive Noise F N H D Y N Y V N { 2 0, } = DHF X + V, V ~ N σ n N = 1 30
The Super-Resolution Problem Y = DHF X + V { 0, 2 } σ, V ~ N n Given Y The measured images (noisy, blurry, down-sampled..) H The blur can be extracted from the camera characteristics D The decimation is dictated by the required resolution ratio F The warp can be estimated using motion estimation σ n The noise can be extracted from the camera / image Recover X HR image 31
The Model as One Equation Y Y1 D1H 1F1 V1 Y V 2 D2H2F2 X + 2 = = = GX M M M Y N DNHNFN V N + V Y G r = resolution factor = 4 MXM = size of the frames = 1000X1000 N = number of frames = 10 of of X, V size size of 2 [ NM 1] 2 2 2 [ NM r M ] 2 2 [ r M 1] size =[10M 1] =[10M 16M] =[16M 1] Linear algebra notation is intended only to develop algorithm 32
SR - Solutions Maximum Lielihood (ML): X = X argmin X X N = 1 N =1 DHF X Y 2 Often ill posed problem! Maximum Aposteriori Probability (MAP) = argmin DHF X Y 2 { } +λa X Smoothness constraint regularization33
ML Reconstruction (LS) ( ) = = N ML Y X X 1 2 2 DHF ε Minimize: ( ) ( ) 0 ˆ 2 1 2 = = = N T T T ML Y X X X DHF H D F ε Thus, require: N T T T N T T T Y X = = = 1 1 ˆ D H F DHF D H F A B B A = Xˆ 34
LS - Iterative Solution Steepest descent Xˆ N T T T n+ = Xˆ β 1 n F H D n = 1 ( DHF Xˆ Y ) Bac projection Simulated error All the above operations can be interpreted as operations performed on images. There is no actual need to use the Matrix-Vector notations as shown here. 35
LS - Iterative Solution Steepest descent ( ) T T T Xˆ n+ = Xˆ 1 n β F H D DHF Xˆ n Y N = 1 For =1..N Xˆ n Y geometry wrap convolve with H down sample - up sample convolve with H T inverse geometry wrap F H D T D T H T F -β Xˆ n+1 36
Example HR image LR + noise X4 Least squares Simulated example from Farisu at al. IEEE trans. On Image Processing, 04 37
Robust Reconstruction Cases of measurement outlier: Some of the images are irrelevant Error in motion estimation Error in the blur function General model mismatch 38
Robust Reconstruction Minimize: ε 2 N ( X ) = = 1 DHF X Y 1 Xˆ N T T T n+ = Xˆ β 1 n F H D sign n = 1 ( DHF Xˆ Y ) 39
Robust Reconstruction Steepest descent For =1..N Xˆ n Xˆ N T T T n+ = Xˆ 1 n β F H D sign n = 1 Y ( DHF Xˆ Y ) geometry wrap convolve with H down sample - sign up sample convolve with H T inverse geometry wrap F H D T D T H T F -β ˆ 40 X n+1
Example - Outliers Least squares HR image LR + noise X4 Simulated example from Farisu at al. IEEE trans. On Image Processing, 04 Robust Reconstruction 41
Example Registration Error L 2 norm based L 1 norm based 20 images, ratio 1:4 42
MAP Reconstruction 2 MAP Regularization term: N = ( X ) DHF X Y λa{ X} ε + =1 2 Tihonov cost function A T { X } = ΓX 2 Total variation A TV { X } = X 1 Bilateral filter A B P P { X } = l= P m= P α l + m X S l x S m y X 1 43
Robust Estimation + Regularization ( ) = = + = + = P P l P P m m y l x m l N X S S X Y X X 1 1 1 2 α λ ε DHF Minimize: ( ) [ ] ( ) + = = = + = + P P l P P m n m y l x n m y l x m l N n T T T n n X S S X S S I Y X X X ˆ ˆ sign ˆ sign ˆ ˆ 1 1 α λ β DHF H D F 44
Example 8 frames Resolution factor of 4 From Farisu at al. IEEE trans. On Image Processing, 04 45
Example Images from Vigilant Ltd. 46
Handling Color in SR 2 MAP N = ( X ) DHF X Y λa{ X} ε + =1 Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr. Better treatment can be obtained if the statistical dependencies between the color layers are taen into account (i.e. forming a prior for color images). In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better. 2 47
SR for Full Color 20 images, ratio 1:4 48
SR+Demosaicing 20 images, ratio 1:4 Mosaiced input Mosaicing and then SR Combined treatment 49
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Special cases and related problems SR in time SR from Examples 50
Special Case Translational Motion In this case H and F commute (bloc circulant): T T HF = F H H F = F T H T Y = = = DHF X + V DF H X + V DF Z + V Z = H X SR is decomposed into 2 steps 1. Find blur HR image from LR images non-iterative 2. Deconvolve the result using H iterative 51
Intuition Y = DF Z + V Z = H X X Z=PSF*X Using the samples can, at most, reconstruct Z To recover X, need to deconvolve Z 52
Step I Find Blurred HR Minimize: ε 2 ML N ( Z) = = 1 DF Z Y 2 L 2 For all frames, copy registered pixels to HR grid and average [Elad & Hel-Or, 01] L 1 For all frames, copy registered pixels to HR grid and use median [Farisu, 04] 53
Solution for L 2 ( ) = = N ML Y Z Z 1 2 2 DF ε Minimize: = = = = N T T N T T Y P 1 1 D F DF D F R Z = P ˆ R ( ) 0 2 = Z Z ε ML Thus, require: Sum of HR grid Diagonal, number Of occurrences per HR grid 54
Step II - Deblur ( ) { } A X Z X X λ ε + = 2 2 H Minimize: ( ) { } + = + n n n T n n X A X Z X X X ˆ ˆ ˆ ˆ ˆ 1 λ β H H 55
Example 64X64 LR 256X256 Before deblur 256X256 After deblur From Pham at al. Proc. Of SPIE-IS&T, 05. Simulated. 56
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Special cases and related problems SR in time SR from Examples 57
Classical Image Super-Resolution Low-resolution images: Scene: High-resolution image: 58
Space-Time Super-Resolution Low-resolution images video sequences: time time time High space-time resolution sequence: time time Super-resolution in space and in time. 59
What is Super-Resolution in Time? Observing events faster than frame-rate. Handles: (1) Motion aliasing (2) Motion blur Application areas: - sports scenes - scientific imaging - etc... 60
(1) Motion Aliasing The Wagon wheel effect: Slow-motion: time time time Continuous signal Sub-sampled in time Slow motion 61
(2) Motion Blur 62
(2) Motion Blur Camera 1: long exposuretime Camera 2: short exposuretime Motion-blur A Spatial artifact caused by Temporal blurring. Operation of camera: exposure time time time 1/frame-rate 1/frame-rate 63
Space-Time Super-Resolution S h (x h,y h,t h ) y t T x x y Blur ernel: PSF l S 1 l S n Exposure time t 64
Example: Motion-Aliasing Input 1 Input 2 Input 3 Input 4 25 [frames/sec] 65
Example: Motion-Aliasing Input sequence in slow-motion (x3): Super-resolution resolution in time (x3): 75 [frames/sec] 75 [frames/sec] 66
Output sequence: Output trajectory: Deblurring: Input: (x15 frame-rate) Without estimating motion of the ball! Output: 3 out of 18 low-resolution input sequences (frame overlays; trajectories): 67
Example: Motion-Blur (real) Frames 4 input at sequences: collision: Video 1 Video 2 Output frame at collision: Video 3 Video 4 68
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Special cases and related problems SR in time SR from Examples 69
Example Based Super Resolution Image is complex for a generative model to model Non-parametric methods Texture synthesis [Efros99],[Feng04] Motion synthesis [Li02] Resolution synthesis
Example Based Super Resolution Generate a high-resolution image from a single low-resolution image, with the help of a set of training images. Slides from Gu, Yu, Tian, Belorylov 2006
Example Based Super Resolution Output HR Input LR Training Set (Low-High resolution image pairs) Training Set
Extensions Super-resolution Through Neighbor Embedding (Manifold learning) 1. For each patch x in image LR image: (a) Find K nearest neighbors in the training set t 1 t. (b) Compute the reconstruction weights w 1 w of the neighbors that minimize the error of reconstructing x. min w1... w (c) Compute HR y as a weighted linear combination of the corresponding high-resolution patches T i using the weights w 1 w. y x = i i w t i w i T i H. Chang, D.Y. Yeung, Y. Xiong, Super-resolution through neighbor embedding, 2004 i 2
Super-resolution Through Neighbor Embedding (Manifold learning,lle) Training Set 3x3 * * * * * w1 + w2 + w3 + w4 + w5 * * * * * w1 + w2 + w3 + w4 + w5 9x9
Super-resolution Through Neighbor Embedding (Manifold learning,lle) Input LR image True HR image Neighborhood Embedding Interpolation
Super-resolution Using High Low Frequency matching pairs Basic idea Divide the image into frequencies (L,M,H) Construct database to store the relations of midfrequency and corresponding high frequency Interpolation for initial magnification Add high frequency details by searching in the training database Freeman, Jones, Pasztor Example Based Super Resolution, 2002
0.0751 0.1238 0.0751 0.1238 0.2042 0.1238 0.0751 0.1238 0.0751 Training phase Blur downsample interpolation High-frequency filter Mid-frequency 1/9 1/9 1/9 1/9 8/9 1/9 1/9 1/9 1/9
Data Set Low resolution patches High resolution patches
Input LR Retrieval phase interpolation filter 1/9 1/9 1/9 1/9 8/9 1/9 1/9 1/9 1/9 Output HR Training Set + Mid-frequency High-frequency
Super-resolution Using High Low Frequency matching pairs Input LR image True HR image High-Low Frequency Interpolation
Super-resolution Using High Low Frequency matching pairs Interpolation Output HR image
Super-resolution Comparison Weighted Manifold restores high res patch by computing weighted combination of high res patches in training set. High-Low Freq Example Based Algorithm finds the best high res patch in Training Set whose low res is most similar to the patch. Weighted Manifold Low-High Freq
Overlapping of Patches For each patch embed the computed high res patch. Use overlapping patches. 1. For each pixel in overlap, compute average over all high res pixels. B1 B2 Overlap Average 2. Use Optimal Cut to combine overlapping patches. B1 B2 Minimal error boundary cut
Overlapping of Patches 3. Grow image and apply neighboring high res constraints. a. Choose best matches for patch and select that which best fits the neighboring constraints. b. Select best matches and use MRF to select patches so that neighboring patches are smooth.
SR by Examples Training set limitation It might seem that SR of an image of one feature (eg, a cat) requires a training set that contains these features (images of other cats). However, this isn t the case.
SR by Examples Training set limitation Although the training set doesn t have to be very similar to the image to be enlarged, it should be in the same image class such as text or color image.
SR from a Single Image SR based on multiple LR images SR based on Database of Images. Can SR be obtained from a single image??
SR from a Single Image Use repetitions in single image.
SR from a Single Image Patches in image are found at different locations and scale. Use different scale to predict what high res of patch should loo lie. Glasner, Bigon & Irani Super resolution from a Single Image, 2010
SR from a Single Image HR from many LR images LR patches induce constraints on HR image. Multi patch LR in single image induces same ind of constraints. Multi scale patch in single image form an Example.
SR from a Single Image Multi Resolution approach combines 2 approaches: 1) Multi LR Images to High res 2) SR based on Examples
SR from a Single Image Glasner, Bigon & Irani Super resolution from a Single Image, 2010
SR from a Single Image Bicubic x3 SR from Single Image x3 Glasner, Bigon & Irani Super resolution from a Single Image, 2010