Physics-based Vision: an Introduction

Transcription

1 Physics-based Vision: an Introduction Robby Tan ANU/NICTA (Vision Science, Technology and Applications) PhD from The University of Tokyo,

2 What is Physics-based? An approach that is principally concerned with the physical (optical) processes that govern the connection between scenes and images. 2

3 Why Physics-Based? The most practical reason: No other approaches could solve some vision problems, without understanding the physics of image formation. 3

4 Other Reasons Images convey optical information. Some information can be obtained by only understanding and solving the physical models. Parameters of physical models are very useful to obtain intrinsic surface properties (which are crucial for recognition). Applications, particularly in computer graphics: rendering realistic images. 4

5 Overview 5

6 Shape from Shading B.K.P. Horn, Robot Vision,

7 Color Constancy 7

8 Highlight Removal Tan et al, Separating Reflection Components of Textured Surfaces, PAMI,

9 Shadow Removal Finlayson et al, On the removal of shadow from images, PAMI,

10 Shadow Removal Application Shadow removal is particularly useful for tracking objects in outdoor scenes. Matsushita et al, Illumination normalization with time-dependent intrinsic images, PAMI,

11 Underwater Schechner et al, Clear underwater vision, CVPR,

12 Bad Weather 12

13 Computer Graphics Applications 13

14 Skin Synthesis Tsumura et al, Image-based skin color and texture analysis, SIGGRAPH,

15 Poisson Image Editing Source images Poisson seamless cloning Perez et al, Poisson image editing, SIGGRAPH,

16 Digital Matting Sun et al, Poisson Matting, SIGGRAPH,

17 Material Editing 17

18 Computer Vision Appearance Underwater Imaging Medical Imaging Computer Graphics Satellite Imaging 18

19 Basic Components of Physics-based Vision 19

20 Reflection light source medium camera object 20

21 Four Basic Components Light Sources Objects Participating Medium Cameras 21

22 Light Source s Properties Color Geometrical properties: distributions, locations, directions Inter-reflection or ambient light properties Light optical effects: Shadow Shading 22

23 Light: Distribution 23

24 Light: Interreflections 24

25 Object s Properties Object 3D geometry: solid and transparent objects Texture Modelling Highlights (or specularity) Object optical parameters: Diffuse, dichomatic: Torrance-Sparrow model Isotropic and anisotropic Translucent objects: subsurface scattering 25

26 Translucent Objects Governed by subsurface scattering theory. 26

27 Isotropic vs Anisotropic 27

28 Participating Medium Atmospheric vision and bad weather Haze, fog, rain, snow, etc. Underwater vision Turbid water Non-turbid water 28

29 Bad Weather 29

30 Cameras/Sensors Polarizing filters Radiometric calibration Beyond visible spectrum Human eyes High Dynamic Range (HDR) images 30

31 Polarization Photography Separating Reflected and Transmitted Scenes 31

32 Polarization and Transparent Object Reconstructing Shape of Transparent Objects 32

33 Radiometric Calibration 33

34 High Dynamic Range (HDR) Images image intensity intensity saturation high dynamic range 0 0 shutter speed shutter speed 34

35 Example of HDR Images 35

36 Basic Flow 36

37 Flow of Physics-based Approach Cameras Optics Mathematics Capture Modeling Interpreting Synthesis Vision Graphics 37

38 Image Formation 38

39 Reflected Light E(λ) S ( x, λ where: E S ( λ ) = ( x, λ ) = illumination spectraldistribution object spectral reflectance I ( x, λ ) = S ( x, λ ) E ( λ ) air ) object λ= wavelength x= { r, s, t } 3D world coordinate 39

40 Spectral Distribution ( E(λ) ) 40

41 Material Types Surface Reflectance S(λ) Homogeneous Objects Inhomogeneous Objects (steels, aluminums, silvers, etc) (plastics, ceramics, acrylics, etc) 41

42 Reflection of Inhomogeneous E(λ) specular I s s x ( x, λ ) = S (, λ ) E ( λ ) ) diffuse I ( x, λ ) = S ( x, λ ) E ( λ I d d x air ( x, λ ) Inhomogeneous objects (plastics, acrylics, etc) S I ( x, λ ) = I d ( x, λ ) + I s ( x, λ ) = ( x ) S ( λ ) E ( λ ) + w ( ) S ( λ ) E ( λ ) = w d d s x s w d x ) S d ( λ ) E ( λ ) + w s ( x ) E ( ( λ ) 42

43 Image Formation E(λ) camera I ( x, λ ) = S ( x, λ ) E ( λ ) R G B = = = I I I ( ( ( x x x,,, λ ) λ ) λ ) q q q R G B ( ( ( air S ( x, λ ) I ( x ) = I ( x, λ ) q ( λ ) d x= { x, y }, λ object the 2D image coordinate λ ) λ ) λ ) d d d λ λ λ 43

44 Camera Sensitivity 44

45 Dichromatic Reflection Model I ( x ) = w ( x ) B ( x ) + w ( x d s ) G where: diffuse specular B ( d x ) x ) = S (, λ ) E ( λ ) q ( λ dλ Ω Ω G = E ( λ ) q ( λ ) dλ w d, w s = the component, geometric respective factor of diffuse and specular reflection ly. 45

46 Chromaticity Image Chromaticity: σ = I I + + I I r g b where: σ { σ r, σ g, { I, I I b} = σ I =, r g b } 46

47 Chromaticity - Examples = = = b g r I I I = = = b g r σ σ σ 47 b = = = b g r I I I = = = b g r σ σ σ

48 Chromaticities Image Chromaticity: σ = I I + I + I r g b I = w B + w s G d Diffuse assume chromaticity: w s = 0 σ σ = w Λ= d = B B r B + w w d d B B B g + w d r + B b + B B r + B g + B b B b b Specular (illumination) chromaticity: Γ= G G r + G g + G b 48

49 Chromaticity-based Model I ( x ) = m ( x ) Λ ( x ) + m ( x d s ) Γ where: Λ=the diffuse chromaticity Γ = the specular chromaticity m + d = w d ( B r + B g B b s = w s G r + G g G m ( + b ) ) B = S d ( λ ) E ( λ ) q ( λ ) dλ Ω G = E ( λ ) q ( λ ) dλ Ω x=the 2D image coordinate 49

50 Color Constancy 50

51 Color Constancy Problem Unknown illumination Canonical Illumination 51

52 Color Constancy Problem I ( x ) = m ( x ) Λ ( x ) + m ( x d x s ) Γ Image intensity (input) Illumination chromaticity 52

53 Chromaticity-Intensity I = m Λ+ m d s Γ σ = I I + I + I r g reflection model Image chromaticity b I = m d ( Λ Γ )( σ σ Γ ) 53

54 Inverse-intensity & Chromaticity Correlation of illumination chromaticity and image chromaticity: image chromaticity σ = p 1 Σ I i +Γ Illumination chromaticity p = m d ( Λ Γ ) I I r + I g + i = I b 54

55 Inverse-intensity Chromaticity σ = p 1 Σ I i +Γ σ c = 1 p + Γ c c Σ I i specular diffuse Independent operation for each color channel c={r, g, b} 55

56 Linear Correlation σ c = p 1 c + Σ I Γ i c 56

57 Multicolored Surfaces σ c = p 1 c + Σ I Γ i c 57

58 Estimating Illumination Two computational steps to estimate illumination chromaticity: Hough Transformation Intersection Counting (Histogram) 58

59 Hough Space σ c = p 1 c + Σ I Γ i c p c P Γ c 59

60 Histogram Analysis Hough Space Chromaticity-count space 60

61 Computational Method Input image inverse intensity chromaticity space Hough space Histogram space Γ c 61

62 Result: a uniform colored surface Estimation: r=0.377 g=0.324 b=0.286 Ground truth (white reference) r=0.371 g=0.318 Solux Halogen Lamp b=

63 Result: multicolored surface Estimation: r=0.319 g=0.438 b=0.212 Ground truth (white reference) r=0.298 g=0.458 b=0.243 Solux Halogen covered by Green Filter 63

64 Result: complex textured surface Estimation: r=0.315 g=0.515 b=0.207 Ground truth (white reference) r=0.282 g=0.481 b=

65 Result: complex scene Lit by fluorescence light in uncontrolled environment Estimation: r=0.321 g=0.346 b=0.309 Ground truth (white reference) r=0.336 g=0.340 b=

66 Accuracy & Robustness The number of experiments: 43 images White reference s standard deviation: 0.01 ~

67 Multicolored Illumination Halogen light Incandescent light Multicolored illumination 67

68 Result: Single Surface Color Input image Estimation: Ground Truth: 68

69 Result: Highly Textured Surface Estimation: Ground Truth: 69

70 Highlight Removal 70

71 Separation Problem b. Diffuse component a. Input c. Specular Component 71

72 Separation Problem I ( x ) = m ( x ) Λ ( x ) + m ( x d s ) Γ Image intensity (input) diffuse specular 72

73 Maximum Chromaticity Chromaticity : Max. Chromaticity : σ= I I + I + I r g b ~ σ = max r ( ) I,, I I r g I + I + g I b b where: I = m d Λ+ m s { σ, σ b} σ =, r g σ Γ { I, I b} I =, r g I 73

74 Max-chromaticity Intensity Space specular diffuse Properties: 1. All diffuse pixels have the same max. chromaticity values 2. Diffuse max. chromaticities are always bigger than those of specular (located at the extreme right side of specular points) 74

75 ~ I = ε Specular-to-diffuse Mechanism Assume: specular component s color is pure white (Γr= Γg= Γb) max( I r, I g, I b ) I + spec ( x 1 ) = m d ( x 1 )Λ m s I spec ( x 1 ) ε I diff ) = m ( )Λ ( x 2 d x 2 I diff ( x 1 ) = a small scalar value ~ σ ~ σ spec diff 75

76 Usage of the Mechanism For synthetic (noise free) image: specular diffuse input Separation result: diffuse component specular component 76

77 Experimental Results a. input b. diffuse component c. specular component 77

78 Experimental Results 78

79 Textured Surface Problem 79

80 Specular-Free Image 80

81 Example in Real Image a. Input image b. Specular-free image 81

82 Separation Framework Input image Specular Reduction Process specular-free image Diffuse-only Verification Process diffuse component specular component All processes solely require two neighboring pixels 82

83 Diffuse-Only Verification Process 83

84 Logarithmic Differentiation Two pixels that have the same geometrical profile but different surface color: I 1 x ) = m d ( x ) ( Λ 1 I 2 x ) = m d ( x ) ( Λ 2 84

85 Logarithmic Differentiation Applying intensity logarithmic differentiation on diffuse pixels, they will produce the same value: Input image (a diffuse pixel): Specular-free image: I 1 ( x ) = m d ( x ) Λ 1 I 2 ( x ) = m d ( x ) Λ 2 ) d d x log I 1 ( x ) = log m d ( x ) + log( Λ 1 d log I 1 ( x ) = log m d ( x ) d x ) d d x log I 2 ( x ) = log m d ( x ) + log( Λ 2 d log I 2 ( x ) = log m d ( x ) d x d d x d log I 1 ( x ) = log I 2 ( x d x ) 85

86 Determining Diffuse Only Pixels Whether two pixels are diffuse is determinable: d ~ d ~ ( x ) = log I ( x ) log I ( x 1 2 d x d x ) ( x ) { = 0 : diffuse 0 : specular or color discontinuity 86

87 Boundary Problem σ g By considering two-neighboring pixels, the problem of color boundary can be solved simply using: σ r ( σ r > thr and σ g > thg) true : false color boundary : specular where: thr, thg = the threshold on red and green channel, respectively 87

88 Diffuse Verification input image Specular-free image Logarithmic Differentiation Logarithmic Differentiation no Boundary? no Identical? yes yes specular diffuse 88

89 Specular Reduction Process 89

90 Chromaticity-Intensity Space a b c a. Specular image b. Spatial Intensity space c. Chromaticity Intensity space 90

91 Iteration Framework 91

92 Separation Framework local (two-neighboring pixels) operation can be done by using the framework: input image Specular Reduction Process specular-free image Diffuse-only Verification Process diffuse component specular component 92

93 Non-White Illumination a single image Illumination chromaticity estimation normalization Iterative-based Separation diffuse component specular component 93

94 Result: a single object Input image Specular-free image 94

95 Separation Result Diffuse reflection component Specular reflection component 95

96 Result: complex scene Input image Specular-free image 96

97 Separation Result: Diffuse reflection component Specular reflection component 97

98 Evaluation object with highlights light source polarizing filter #1 camera polarizing filter #2 resulted image 98

99 Evaluation 1: a. Input b. Result using polarizing filters b. Estimated diffuse reflection 99

100 Evaluation 1: error Input-polarization comparison Estimated diffusepolarization red channel green channel blue channel 100

101 Evaluation 2 a. Input b. Result using polarizing filters b. Estimated diffuse reflection 101

102 Evaluation 2: error red channel green channel blue channel 102

103 Conclusion Color Constancy Reflections Separation a. Input b. Normalized image c. Diffuse component c. Specular Component 103

104 Image-based Skin Color and Texture Analysis (siggraph 03) N. Tsumura, N. Ojima, K. Sato, etc. Chiba University and Kao Corporation 104

105 General Framework 105

106 Subsurface Scattering 106

107 Imaging Model Two types of reflections: Interface reflection (specular reflection) Body reflection: Epidermis layer (melanin) Dermis layer (hemoglobin) The body reflection follows Lambert-Beer law 107

108 Lambert-Beer Law where: ρ m, ρ h = the pigment density σ m, σ h = the spectral cross - sections, = the mean path lengths of l m l h photons in the dermis and epidermis layers Reflection model for image intensities by assuming narrow-band sensor sensitivity: 108

109 Logarithmic Model where: 109

110 Extracting Skin Color 110

111 Find Skin Color without Shading 111

112 Remove Shading by Projection 112

113 Results: Skin Color Synthesis 113

114 Results: Alcohol and Tanning 114

115 Recent Research 115

116 Current Research 1 116

117 Current Research 2 117

118 Visual Ambiguities 118

119 Gray? 119

120 Blue and yellow! 120

121 Orange and purple? 121

122 Red! 122

123 Contact anu.edu.au users.rsise.anu.edu.au/~robby/ 123