Colorization: History

Transcription

1 Colorization

2 Colorization: History Hand tinting

3 Colorization: History Film colorization Colorization in 1986 Colorization in 2004

4 Overview Colorization by example Colorization using scribbles

5 Transferring Color to Greyscale Images T. Welsh, M. Ashikhmin, and K. Mueller SIGGRAPH 2002

6 The Basic Approach Convert source image to decorrelated lαβ color space l: luminance α, β: chromatic channels (yellow/blue and red/green) Perform luminance remapping (histogram matching) Take ~200 color samples from the source image For each pixel in the target image (in scanline order): Find best matching source pixel (compare luminance and std. dev. of luminance values in neighborhood) Transfer color from source pixel to target pixel source target result

7 Recall: Image Analogies

8 Problem Global procedure fails when corresponding colors don t have corresponding luminance values Source image Target image Colorized target Grayscale source image

9 Solution The user specifies corresponding swatches in the source and target images

10 Colorization with swatches Transfer between swatches Global transfer Extend to the rest of image

11 Colorization with swatches: Details Transfer color from source to target swatches Perform luminance remapping between corresponding swatches Take ~50 samples from each source swatch Extend colorized swatches to the rest of image For each grayscale pixel, find best matching pixel in a colorized swatch in the target image Matching function is SSD of grayscale neighborhoods Transfer color from matching pixel to grayscale pixel

12 Example results

13 Example results

14 Example results: Scientific visualization

15 Video colorization First transfer color between swatches for a single frame Use the colorized swatches in the single frame to transfer color to the rest of the sequence

16 Video colorization results

17 Video colorization results

18 Brain volume colorization

19 Discussion of implementation choices Effect of color space Sampling scheme for source pixels Matching function between source and target pixels Additional constraints for search (i.e., spatial coherence) Selection of sample image

20 Colorization Using Optimization A. Levin, D. Lischinski, Y. Weiss SIGGRAPH

21 Overview Input: grayscale image with color scribbles Output: Colorized image

22 The Approach Two neighboring pixels r, s should have similar colors if their intensities are similar The goal is to minimize the difference between the color U(r) at pixel r and the weighted average of the colors at neighboring pixels

23 Objective function sum over all pixels color of pixel r sum over pixels in the neighborhood of r affinity between r and s color of pixel s

24 Objective function Possible affinity functions: Neighborhood definition: for video, take optical flow into account Constraints: color of user-specified pixels remains fixed Optimization: sparse linear system

25 Results Colors from the original image used for the scribbles Processing time: ~15sec/frame

26

27

28

29 Comparison to segmentation-based colorization Segmented image Segmentation + flood fill Colorization by optimization

30 Recoloring Original image Mask and scribbles Final image

31

32 More recoloring Original image Scribbles Final image

33 Progressive colorization

34 Video colorization Grayscale video Input scribbles

35 Video colorization Grayscale video Colorized video

36 Video colorization Grayscale video Input scribbles

37 Video colorization Grayscale video Colorized video

38 Video colorization Grayscale video Input scribbles

39 Video colorization Grayscale video Colorized video

40 Colorization by Example R. Irony, D. Cohen-Or, and D. Lischinski Eurographics Symposium on Rendering, 2005

41 Motivation Improve spatial consistency of examplebased transfer methods such as Welsh et al. (2002) Reduce the amount of manual supervision of scribble-based methods such as Levin et al. (2004)

42 The importance of spatial coherence Source image Target image Image colorized by method of Welsh et al.

43 The importance of spatial coherence Source image Target image Proposed method

44 Overview of approach

45 Example result Reference (source) image Automatic segmentation Target image Pixel classification Smoothed pixel classification Colorized target

46 Example result Reference (source) image Manual segmentation Target image Automatic classification Colorized target image

47 Manual vs. automatic segmentation Source image Manual segmentation Automatic segmentation Colorization Target image Classification based on manual segmentation Classification based on automatic segmentation

48 Colorizing multiple frames

49 Natural Image Colorization L. Qing, F. Wen, D. Cohen-Or, L. Liang, Y.-Q. Xu, H. Shum Eurographics Symposium on Rendering, 2007

50 Motivation Reduce the amount of user interaction necessary to produce complex, nuanced color images Handle highly textured images non-adjacent regions of similar texture Colorization by optimization (Levin et al.)

51 Motivation Reduce the amount of user interaction necessary to produce complex, nuanced color images Handle highly textured images non-adjacent regions of similar texture Proposed method

52 Outline of method 1. The user draws strokes indicating regions that (roughly) share the same color 2. Strokes are used for automatic texture segmentation 3. The user selects color for a few pixels in each region 4. Color is transferred automatically based on segmentation and selected colors

53 Segmentation Iterative process: propagate labels to regions similar in intensity and texture, but not necessarily spatially contiguous

54 Color mapping Piecewise-linear interpolation of selected colors inside each region Soft blending of colors around the region boundaries

55 Comparison Levin result 1 Levin result 2 Proposed method

56 Comparison Proposed method Levin et al.

57 More results

58 More results

59 More results

60 Difficult example

61 Closeup

62 Colorization: Summary Example-based methods Transferring color to grayscale images (Welsh et al. 2002) Shortcoming: spatial coherence Colorization by example (Irony et al. 2005) Spatially coherent texture segmentation Stroke-based methods Colorization using optimization (Levin et al. 2004) Shortcomings: color leaking, too many strokes required for textured images Natural image colorization (Qing et al. 2007) Handle images with non-contiguous textures