Image Compositing and Blending

Transcription

1 Computational Photography and Capture: Image Compositing and Blending Gabriel Brostow & Tim Weyrich TA: Frederic Besse

2 Vignetting 3 Figure from

3 Radial Distortion Magnification/focal length different for different angles of inclination pincushion (tele-photo) barrel (wide-angle) 4 Can be corrected! (if parameters are know)

4 Ultra wide-angle optics Sometimes distortion is what you want Fisheye lens Cata-dioptric system (lens + mirror) 5

5 Week Date Topic Hours 1 12-Jan Introduction to Computational Photography and Capture Jan Intro + More on Cameras, Sensors and Color Jan No lecture! (Go capture bracketed photos?) Jan Blending, Compositing, Poisson Editing Jan Seam Carving and Time-Lapse Jan Warping, Morphing, Mosaics and Panoramas Feb High-Dynamic-Range Imaging and Tone Mapping Feb Hybrid Images, Flash and Multi-Flash Photography Feb Colourisation and Colour Transfer Feb Image Inpainting and Texture Synthesis Feb Video Based Rendering of Scenes I Feb Video Based Rendering of Scenes II Mar Video Texture Synthesis Mar Video Sprites Mar Deblurring/Dehazing and Coded Aperture Imaging Mar Image-based Rendering Mar Motion Capture guest lecture by Doug Griffin Mar Capturing Geometry with Active Lighting Mar Intrinsic Images Mar Dual Photography and Reflectance Analysis 2

6 Image Compositing and Blending NASA Slides from Alexei Efros

7 Image Compositing Aside: Play the what s fake? game

8 Section of Photo Clip Art Results

9 Section of Photo Clip Art Results

10

11

12

13

14

15

16

17 Photo Clip Art, Lalonde at al. 2007: Web, Video

18 Compositing Procedure 1. Extract Sprites (e.g. using Intelligent Scissors in Photoshop) 2. Blend them into the composite (in the right order) Composite by David Dewey

19 Just replacing pixels rarely works Binary mask Problems: boundries & transparency (shadows)

20 Two More Problems: Semi-transparent objects Pixel spacing too large

21 Solution: alpha channel Add one more channel: Image(R,G,B,alpha) Encodes transparency (or pixel coverage): Alpha = 1: opaque object (complete coverage) Alpha = 0: transparent object (no coverage) 0<Alpha<1: semi-transparent (partial coverage) Example: alpha = 0.3 Partial coverage or semi-transparency

22 Alpha Blend Here I comp = I α I fg + (1- I α )I bg alpha mask shadow

23 Alpha Blend Here I comp = I fg + (1- )I bg alpha mask shadow

24 Multiple Alpha Blending So far we assumed that one image (background) is opaque. If blending semi-transparent sprites (the A over B operation): I comp = ai a + (1- a) bi b comp = a + (1- a) b Note: sometimes alpha is premultiplied: im( R, G, B, ): I comp = I a + (1- a)i b (same for I b!)

25 No simple physical interpretation, but it smoothes the seams Alpha Hacking

26 Feathering Encoding as transparency = I blend = I left + (1- )I right

27 Setting alpha: simple averaging Alpha =.5 in overlap region

28 Setting alpha: center seam Distance transform Alpha = logical(dtrans1>dtrans2)

29 Setting alpha: blurred seam Alpha = blurred

30 Setting alpha: center weighting Distance transform Alpha = dtrans1 / (dtrans1+dtrans2) Ghost!

31 Effect of Window Size 1 left 1 0 right 0

32 Effect of Window Size

33 Good Window Size 1 0 Optimal Window: smooth but not ghosted

34 What is the Optimal Window? To avoid seams window = size of largest prominent feature To avoid ghosting window <= 2*size of smallest prominent feature Natural to cast this in the Fourier domain largest frequency <= 2*size of smallest frequency image frequency content should occupy one octave (power of two) FFT

35 What if the Frequency Spread is Wide FFT Idea (Burt and Adelson) Compute F left = FFT(I left ), F right = FFT(I right ) Decompose Fourier image into octaves (bands) F left = F left1 + F left2 + Feather corresponding octaves F left i with F right i Can compute inverse FFT and feather in spatial domain Combine feathered octave images in frequency domain? Better implemented in spatial domain

36 Octaves in the Spatial Domain Lowpass Images Bandpass Images

37 Pyramid Blending Left pyramid blend Right pyramid

38 Pyramid Blending

39 laplacian level 4 laplacian level 2 laplacian level 0 left pyramid right pyramid blended pyramid

40 Laplacian Pyramid: Blending General Approach: 1. Build Laplacian pyramids LA and LB from images A and B 2. Build a Gaussian pyramid GR from selected region R 3. Form a combined pyramid LS from LA and LB using nodes of GR as weights: LS(i,j) = GR(I,j,)*LA(I,j) + (1-GR(I,j))*LB(I,j) 4. Collapse the LS pyramid to get the final blended image

41 Blending Regions

42 Horror Photo david martin (Boston College)

43 Results from CMU class (fall 2005) Chris Cameron

44 Season Blending (St. Petersburg)

45 Season Blending (St. Petersburg)

46 Simplification: Two-band Blending Brown & Lowe, 2003 Only use two bands: high freq. and low freq. Blends low freq. smoothly Blend high freq. with no smoothing: use binary alpha

47 High frequency ( < 2 pixels) 2-band Blending Low frequency ( > 2 pixels)

48 Linear Blending

49 2-band Blending

50 Gradient Domain In Pyramid Blending, we decomposed our image into 2 nd derivatives (Laplacian) and a low-res image Let us now look at 1 st derivatives (gradients): No need for low-res image captures everything (up to a constant) Idea: Differentiate Blend Reintegrate

51 Gradient Domain blending (1D) Two signals bright dark Regular blending Blending derivatives

52 Gradient Domain Blending (2D) Trickier in 2D: Take partial derivatives dx and dy (the gradient field) Fidle around with them (smooth, blend, feather, etc) Reintegrate But now integral(dx) might not equal integral(dy) Find the most agreeable solution Equivalent to solving Poisson equation Can use FFT, deconvolution, multigrid solvers, etc.

53 Perez et al., 2003

54 Perez et al, 2003 Limitations: editing Can t do contrast reversal (gray on black -> gray on white) Colored backgrounds bleed through Images need to be very well aligned

55 Interactive Gradient Domain Real-Time Gradient-Domain Painting <video>

56 More to Say About Matting Image and Video Matting: A Survey (link) (race) Jue Wang and Michael Cohen. Image and Video Matting: A Survey. Foundations and Trends in Computer Graphics and Vision, Vol. 3, No.2, 2007.

57 Don t blend, CUT! Moving objects become ghosts So far we only tried to blend between two images. What about finding an optimal seam?

58 Davis, 1998 Segment the mosaic Single source image per segment Avoid artifacts along boundries Dijkstra s algorithm

59 Minimal error boundary overlapping blocks vertical boundary 2 _ = overlap error min. error boundary

60 Graphcuts What if we want similar cut-where-thingsagree idea, but for closed regions? Dynamic programming can t handle loops Code is online

61 Graph cuts (simple example à la Boykov&Jolly, ICCV 01) hard constraint n-links t a cut s hard constraint Minimum cost cut can be computed in polynomial time (max-flow/min-cut algorithms) Slide from Yuri Boykov

62 Seam Optimization A B Construct graph such that: Graph Mincut Best Seam Slide courtesy Vivek Kwatra

63 Seam Optimization A B Construct graph such that: Graph Mincut Best Seam Slide courtesy Vivek Kwatra

64 Seam Optimization B A Potential Seam Cost Constraint A B Slide courtesy Vivek Kwatra

65 Seam Optimization A B Seam A B Slide courtesy Vivek Kwatra

66 Lazy Snapping SIGGRAPH 2004 Interactive segmentation using graphcuts

67 Putting it all together Compositing images/mosaics Have a clever blending function Feathering Center-weighted blend different frequencies differently Gradient based blending Choose the right pixels from each image Dynamic programming optimal seams Graph-cuts Now, let s put it all together: Interactive Digital Photomontage, 2004 (video)

68

69

70 Matting for Video Videos! VideoSnapCut.mp4