Sky is Not the Limit: Semantic-Aware Sky Replacement

Transcription

1 Sky is Not the Limit: Semantic-Aware Sky Replacement ACM Transactions on Graphics (SIGGRAPH), 2016 Yi-Hsuan Tsai UC Merced Xiaohui Shen Adobe Research Zhe Lin Adobe Research Kalyan Sunkavalli Adobe Research Ming-Hsuan Yang UC Merced 1

2 Input image 2

3 Input image 3

4 Input image Our result Automatic 4

5 Input image Diverse & Realistic! Automatic 5

6 6

7 7

8 8

9 9

10 Overview Input image Sky Segmentation Sky Search Sky Replacement Output images 10

11 Previous work: sky segmentation Sky/non-sky classifier [Tao et al. SIGGRAPH 09] Scene parsing [Long et al. CVPR 15] Input image [Long et al. CVPR 15] 11

12 Previous work: sky image search GIST [Hays and Efros SIGGRAPH 07, Liu et al. CGF 14] Limitation: global scene layout Input image Reference image 1 Reference image 2 12

13 Previous work: appearance transfer Global transfer [Reinhard et al. 2001, Tao et al. SIGGRAPH 09] Local transfer [Wu et al. CGF 13, Laffont et al. SIGGRAPH 14] Input image Input image with replaced sky 13

14 Previous work: appearance transfer Global transfer [Reinhard et al. 2001, Tao et al. SIGGRAPH 09] Local transfer [Wu et al. CGF 13, Laffont et al. SIGGRAPH 14] Input image Global transfer 14

15 Key idea: semantic guidance Input image Sky Segmentation Sky Search Sky Replacement Semantic Scene Parsing 15

16 Semantic Scene Parsing [Long et al. CVPR 15] Pixel-wise segmentation Semantic response Scene Parsing Sky Tree Building Fg Road Semantic Response... Sky Building Road 16

17 Input image Sky Segmentation Sky Search Sky Replacement Semantic Scene Parsing 17

18 Sky Segmentation Input image Scene Parsing Online Refinement Online model for sky Color, texture, semantics Better than dense CRF Alpha matte 18

19 Sky Segmentation Results 19

20 Input image Results from scene parsing Our refined results 20

21 21

22 22

23 Comparison to DeepLab [Chen et al. 2015]

24 Input image Sky Segmentation Sky Search Sky Replacement Semantic Scene Parsing 28

25 Sky Image Search Sky Image Database (415 Images) Input image 29

26 Sky Image Search Sky Image Database (415 Images) Input image Sky Search Semantic layout descriptor Account for local content 30

27 Semantic Layout Descriptor Input image Sky Building Road... Semantic responses Pixel-wise responses Normalize: from 0 to 1 31

28 Semantic Layout Descriptor Input image Sky Building Road... Average pooling Global info Local content Descriptor... 32

29 Input image Sky Segmentation Sky Search Sky Replacement Semantic Scene Parsing 33

30 Sky Replacement Input image Outputs Sky Alignment Semantic-aware Transfer Reference images Semantic-aware Transfer Adjust foreground appearance Account for semantic regions 34

31 Semantic-aware Transfer Propose a soft mapping method Weighted transfer for category n on pixel x Sky Building Road Pixel x T(x) = 0.45*T sky (x) *T bld (x) *T road (x) 35

32 Semantic-aware Transfer Transfer Function T n for category n Transfer luminance and color Not all the semantic regions are matched! T 1 (x) T 2 (x)? Color Matched Use chrominance Non-matched Use color temperature 36

33 Semantic-aware Transfer Input image Scene parsing Our result 37

34 Sky Replacement Results 38

35 39

36 Input image 40

37 41

38 42

39 43

40 44

41 Sky Replacement with User Preference 45

42 Input image 46

43 Input image 47

44 Input image Preferred sky 48

45 Performance Evaluation 49

46 Comparisons of different transfer methods Input image Reference [Tao, et al. 2009] Ours (w/o semantics) Our method 50

47 Comparisons of different transfer methods Input image Reference [Tao, et al. 2009] Ours (w/o semantics) Our method 51

48 More realistic 52

49 Comparisons of different search methods Random selection Input image Our method 53

50 Comparisons of different search methods Random selection Input image Our method 54

51 Comparisons of different search methods GIST based method Input image Our method 55

52 More realistic and interesting 56

53 Limitation Input image Output image 57

54 Conclusions Project website Automatic sky replacement Realistic results Semantics helps a lot Sky segmentation Sky image search Appearance transfer 58