Last Lecture Edge Detection Filtering Pramid
Toda Motion Deblur Image Transformation
Removing Camera Shake from a Single Photograph Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis and William T. Freeman http://people.csail.mit.edu/fergus/research/deblur.html Massachusetts Institute of Technolog and Universit of Toronto
Overview Original Our algorithm
Close-up Original Naïve Sharpening Our algorithm
Let s take a photo Blurr result
Slow-motion repla
Slow-motion repla Motion of camera
Image formation process = Blurr image Sharp image Blur kernel Input to algorithm Model is approimation Desired output Convolution operator
Wh is this hard? Simple analog: is the product of two numbers. What are the? No unique solution: = = 2 5.5 = 3 3.667 etc.. Need more information!!!!
Multiple possible solutions Sharp image Blur kernel = Blurr image = =
Natural image statistics Characteristic distribution with heav tails Histogram of image gradients
Blur images have different statistics Histogram of image gradients
Parametric distribution Histogram of image gradients Use parametric model of sharp image statistics
Three sources of information. Reconstruction constraint: = Estimated sharp image Estimated blur kernel Input blurr image 2. Image prior: 3. Blur prior: Distribution of gradients Positive & Sparse
Variational Baesian method Based on work of Miskin & Macka 2 Keeps track of uncertaint in estimates of image and blur b using a distribution instead of a single estimate Helps avoid local maima and over-fitting
Score Variational Baesian method Objective function for a single variable Maimum a-posteriori (MAP) Variational Baes Piel intensit
Overview of algorithm. Pre-processing Input image 2. Kernel estimation - Multi-scale approach 3. Image reconstruction - Standard non-blind deconvolution routine
Preprocessing Input image Convert to grascale Remove gamma correction User selects patch from image Baesian inference too slow to run on whole image Infer kernel from this patch
Initialization Input image Convert to grascale Remove gamma correction User selects patch from image Initialize 33 blur kernel Blurr patch Initial image estimate Initial blur kernel
Inferring the kernel: multiscale method Input image Convert to grascale Remove gamma correction Loop over scales User selects patch from image Upsample estimates Variational Baes Initialize 33 blur kernel Use multi-scale approach to avoid local minima:
Image Reconstruction Input image Convert to grascale Remove gamma correction Loop over scales User selects patch from image Full resolution blur estimate Upsample estimates Variational Baes Initialize 33 blur kernel Non-blind deconvolution (Richardson-Luc) Deblurred image
Results on real images Submitted b people from their own photo collections Tpe of camera unknown Output does contain artifacts Increased noise Ringing Compares well to eisting methods
Original photograph
Blur kernel Our output
Original photograph Matlab s deconvblind
Original photograph
Matlab s deconvblind
Photoshop sharpen more
Our output Blur kernel
Original photograph
Our output Blur kernel
Original photograph
Our output Blur kernel
Matlab s deconvblind
Original photograph
Our output Blur kernel
Close-up of bird Original Our output
Original photograph
Blur kernel Our output
Image artifacts & estimated kernels Blur kernels Image patterns Note: blur kernels were inferred from large image patches, NOT the image patterns shown
Summar Method for removing camera shake from real photographs First method that can handle complicated blur kernels Uses natural image statistics Non-blind deconvolution currentl simplistic
Image Warping image filtering: change range of image g() = T(f()) f T f image warping: change domain of image g() = f(t()) f T f
Image Warping image filtering: change range of image g() = T(f()) T image warping: change domain of image g() = f(t()) T
Parametric (global) warping Eamples of parametric warps: translation rotation aspect affine perspective clindrical
Parametric (global) warping T p = (,) p = (, ) Transformation T is a coordinate-changing machine: p = T(p) What does it mean that T is global? Is the same for an point p can be described b just a few numbers (parameters) Let s represent T as a matri: p = Mp ' ' M
Scaling Scaling a coordinate means multipling each of its components b a scalar Uniform scaling means this scalar is the same for all components: 2
Scaling Non-uniform scaling: different scalars per component: X 2, Y.5
Scaling Scaling operation: Or, in matri form: b a ' ' b a ' ' scaling matri S What s inverse of S?
2-D Rotation (, ) (, ) = cos() - sin() = sin() + cos()
2-D Rotation This is eas to capture in matri form: ' ' cos sin sin R cos Even though sin() and cos() are nonlinear functions of, is a linear combination of and is a linear combination of and What is the inverse transformation? Rotation b For rotation matrices T R R
22 Matrices What tpes of transformations can be represented with a 22 matri? 2D Identit? ' ' ' ' 2D Scale around (,)? s s * ' * ' s s ' '
22 Matrices What tpes of transformations can be represented with a 22 matri? 2D Rotate around (,)? * cos * sin ' * sin * cos ' cos sin sin cos ' ' 2D Shear? sh sh * ' * ' sh sh ' '
22 Matrices What tpes of transformations can be represented with a 22 matri? 2D Mirror about Y ais? ' ' ' ' 2D Mirror over (,)? ' ' ' '
22 Matrices What tpes of transformations can be represented with a 22 matri? 2D Translation? ' t ' t NO! Onl linear 2D transformations can be represented with a 22 matri
All 2D Linear Transformations Linear transformations are combinations of Scale, Rotation, ' Shear, and Mirror ' Properties of linear transformations: Origin maps to origin Lines map to lines Parallel lines remain parallel Ratios are preserved Closed under composition ' ' a c be d g a c b d f i h k j l
Homogeneous Coordinates Q: How can we represent translation as a 33 matri? ' t ' t
Homogeneous Coordinates Homogeneous coordinates represent coordinates in 2 dimensions with a 3-vector
Homogeneous Coordinates Q: How can we represent translation as a 33 matri? A: Using the rightmost column: t t Translation t t ' '
Translation Eample of translation ' ' t t t t t = 2 t = Homogeneous Coordinates
Homogeneous Coordinates Add a 3rd coordinate to ever 2D point (,, w) represents a point at location (/w, /w) (,, ) represents a point at infinit (,, ) is not allowed 2 (2,,) or (4,2,2) or (6,3,3) Convenient coordinate sstem to represent man useful transformations 2
Basic 2D Transformations Basic 2D transformations as 33 matrices cos sin sin cos ' ' ' ' t t ' ' sh sh Translate Rotate Shear ' ' s s Scale
Affine Transformations Affine transformations are combinations of Linear transformations, and Translations Properties of affine transformations: Origin does not necessaril map to origin Lines map to lines Parallel lines remain parallel Ratios are preserved Closed under composition w f e d c b a w ' '
Projective Transformations Projective transformations Affine transformations, and Projective warps Properties of projective transformations: Origin does not necessaril map to origin Lines map to lines Parallel lines do not necessaril remain parallel Ratios are not preserved Closed under composition w i h g f e d c b a w ' ' '
Matri Composition Transformations can be combined b matri multiplication w s s t t w cos sin sin cos ' ' ' p = T(t,t ) R() S(s,s ) p
2D image transformations These transformations are a nested set of groups Closed under composition and inverse is a member
Recovering Transformations? T(,) f(,) g(, ) What if we know f and g and want to recover the transform T? Using correspondences How man do we need?
Translation: # correspondences? How man correspondences needed for translation? How man Degrees of Freedom? What is the transformation matri? T(,)? ' ' p p p p M
Euclidian: # correspondences?? T(,) How man correspondences needed for translation+rotation? How man DOF?
Affine: # correspondences?? T(,) How man correspondences needed for affine? How man DOF?
Projective: # correspondences?? T(,) How man correspondences needed for projective? How man DOF?
Eample: warping triangles Given two triangles: ABC and A B C in 2D (2 numbers) Need to find transform T to transfer all piels from one to the other. What kind of transformation is T? How can we compute the transformation matri: T(,)? A B C A C B Source Destination ' ' f e d c b a
warping triangles (Barcentric Coordinaes) (,) A B Source T (,) (,) Inverse change of basis change of basis C A T 2 B Destination Don t forget to move the origin too! C Ver useful in Graphics
Image morphing The goal is to snthesize a fluid transformation from one image to another. Cross dissolving is a common transition between cuts, but it is not good for morphing because of the ghosting effects. image # dissolving image #2
Image morphing Wh ghosting? Morphing = warping + cross-dissolving shape (geometric) color (photometric)
Image morphing image # cross-dissolving image #2 warp morphing warp
Morphing sequence
Warp b triangulation
Image warping T(,) f(,) g(, ) Given a coordinate transform (, ) = T(,) and a source image f(,), how do we compute a transformed image g(, ) = f(t(,))?
Forward warping T(,) f(,) g(, ) Send each piel f(,) to its corresponding location (, ) = T(,) in the second image Q: what if piel lands between two piels?
Forward warping T(,) f(,) g(, ) Send each piel f(,) to its corresponding location (, ) = T(,) in the second image Q: what if piel lands between two piels? A: distribute color among neighboring piels (, ) Known as splatting
Inverse warping T - (,) f(,) g(, ) Get each piel g(, ) from its corresponding location (,) = T - (, ) in the first image Q: what if piel comes from between two piels?
Inverse warping T - (,) f(,) g(, ) Get each piel g(, ) from its corresponding location (,) = T - (, ) in the first image Q: what if piel comes from between two piels? A: Interpolate color value from neighbors nearest neighbor, bilinear, Gaussian, bicubic