Today. Motivation. Motivation. Image gradient. Image gradient. Computational Photography

Transcription

1 Computational Photography Matthias Zwicker University of Bern Fall 009 Today Gradient domain image manipulation Introduction Gradient cut & paste Tone mapping Color-to-gray conversion Motivation Cut & paste Motivation Cut & paste Cut & paste Cut & paste Gradient cut & paste User input User input Image gradient Gradient: vector of partial derivatives of multivariate function Image gradient à f f(x, y) =, f! x y Vector valued function (vector field) of two variables, like image itself Also called gradient field Will treat color channels separately Image corresponds to three D functions Each has its own gradient Image gradient Discretization with finite differences f f f(x +1,y) f(x, y) x Visualization f(x, y +1) f(x, y) y 1

2 Image gradient Gradient vector points in direction of largest change Length indicates rate of change Image gradient If we know gradient of image, can we reconstruct image itself? Is any D vector field a gradient field of some image? Image gradient M. C. Escher, Ascending and descending Penrose stairs Today Gradient domain image manipulation Introduction Gradient cut & paste Tone mapping Color-to-gray conversion Gradient cut & paste Cut & paste gradients instead of pixel values Reconstruct image with gradients as similar as possible to result of cut & paste Mathematical formulation Optimization problem: Find image f, such that Its gradients are as similar as possible to given gradient field v Satisfies boundary conditions known unknown Cut gradients from source Paste into destination Reconstruct image Guidance field Destination, boundary conditions

3 1D function Matrix formulation f(x) 1 n/a 4 1 n/a x Gradient cut & paste f(x) x Desired gradients 1 n/a ? 1 5? 6? f System of equations to reconstruct function 1 n/a ? 1 5? 6? Remove unnecessary equations 3 4 N/A 4? 1 5? 6? Remove unnecessary equations 3 4 N/A 4? 1 5? L f f Overdetermined system of equations Minimize Normal equations 6?

4 System of equations Normal equations Remember Note d f 1f(x 1) + f(x, y) 1f(x +1) dx Second derivative! Derivative of first derivative = second derivative! Mathematical formulation Original formulation Equivalent Euler-Lagrange equation Also known as Poisson equation Remember Laplace operator 4f = f = f x + f y f = f(x +1,y)+f(x 1,y)+f(x, y +1)+f(x, y 1) 4f(x, y) Divergence div v = vx x + v y y Euler-Lagrange equation Calculus of variations Study functionals, instead of functions Find stationary points (extrema) Minimal surfaces Fermat s principle: light takes path that is stationary point of optical length Euler-Lagrange equation Problem: Get function q that is stationary point (minimum, maximum) of functional L is real valued function, differentiable Solution satisfies Euler-Lagrange equation L x, L v partial derivatives wrt nd and 3rd argument Proof, more details Euler-Lagrange equation Functionals using functions of several variables Discretization in D Based on explicit formulation Linear least squares problem Euler-Lagrange equations x or y component of discrete gradient q q p q q Pixel neighborhood x or y component of guidance fields known unknown 4

5 Discretization in D Solution of linear least squares problem given by set of linear equations Pixel neighborhood N p p N p =4 known unknown Linear system of equations Number of unknown pixels is large! Up to millions of unknowns But system is sparse Most coefficients are zero Many efficient algorithms available Iterative solution Matlab provides numerous implementations Gauss-Seidel method System of equations Iteration Solve equations i for unknown x i in each step Cycle throgh equations until convergence Gauss-Seidel method Pseudocode No need to store matrix explicitly choose an initial approximation x = (x 1,, x n ) do until convergence: for i := (1,, n): σ = 0 for j := (1,, i-1, i+1,, n): σ = σ + a ij * x j next j x i = (b i - σ) / a ii next i next do Converges for positive definite matrices Alternative to determine desired gradients gradient mixing Cut & paste Gradient cut & paste Source/destination ( seamless cloning ) f g Reconstructed image 5

6 Local gamma compression of gradient f Output Summary Poisson image editing Select desired gradients (guidance field) Specify boundary conditions Assemble system of equations Solve known unknown Guidance field Destination, boundary conditions Notes Apply to color channels separately Can lead to color shifts Working in log domain useful Doesn t work well if background of combined images is very different Be careful about boundaries Improvement cuhk edu hk/~leojia/all project webpages/ddp/dragdroppasting pdf Boundary problems Frédo Durand Gradient domain tone mapping Fattal et al., Gradient Domain High Dynamic Range Compression, SIGGRAPH 00 Observation: large changes in luminance correspond to large gradients Idea 1. Identify large gradients. Attenuate their magnitude (but keep direction) 3. Reconstruct image from modified gradients Note: work in log-domain Gradients in log-domain correspond to ratios of luminance Local contrast Gradient domain tone mapping Visualization using one scanline Horizontal scale not consistent HDR input Log domain Gradients Attenuated Reconstruction LDR output gradients (log domain) Gradient attenuation Multiply gradients with attenuation function Attenuated gradients denoted G Gradient attenuation function Gradients of magnitude unchanged Gradients larger than attenuated, given < 1 To capture edges on all scales, compute gradients at different resolutions k Final attenuation factor is multiplication over all resolutions 6

7 Reconstruction Solve poisson problem I =divg Unknown image I Attenuated gradients G Laplacian I = I x + I y Divergence div G Gx x Gy x + y y Neumann Boundary conditions Directional derivative perpendicular to boundary is zero I n =0 Note: compression & reconstruction operates on luminance only Gradient attenuation function Input Gradient attenuation function Color to gray conversion Gooch et al., ColorGray: Salience- Preserving Color Removal, SIGGRAPH Observation: luminance based conversion may miss perceptually important detail Ward et al Gradient domain tone mapping Input Photoshop conversion (005) Salience preservation Approach 1. Convert image to perceptually uniform color space (CIE L*a*b*). Extract salience from chrominance and luminance differences 3. Reconstruct grayscale image that matches target differences Extracting salience Salience represented as target difference ij between pixels i and j Function of both luminance and chrominance differences Details see paper 7

8 Reconstruction Find grayscale pixel values g i by minimizing Difference in grayscale value corresponds to target t difference as much as possible Similar to Poisson equation, but not same! Paper proposes to use all pixel pairs i,j in image I.e., K contains all image pixels Input Photoshop colorgray Input Photoshop colorgray Conclusions Various image manipulation tasks achieved by modifying image gradients Copy & paste, tone mapping, color-to-gray Advantage: specify desired relations (differences) e between pixels, instead of pixel values Need to reconstruct output from modified gradients Poisson equation, or equivalently, linear least squares problem Efficient algorithms exist Next time More image manipulation using optimization 8