CSCI 1290: Comp Photo

Size: px

Start display at page:

Download "CSCI 1290: Comp Photo"

Marlene Powers
5 years ago
Views:

1 CSCI 1290: Comp Photo Fall Brown University James Tompkin Many slides thanks to James Hays old CS 129 course, along with all of its acknowledgements.

2 Feedback from Project 0 MATLAB: Live Scripts!= Scripts *.mlx vs. *.m Live scripts like Jupyter notebook Live scripts seem to take a really long time to compute.

3 Project 1 code How are people getting along? What issues are you facing? (Tell me now.)

4 High Dynamic Range Images Slides from Paul Debevec and Alexei A. Efros

5 Problem: Dynamic Range The real world is high dynamic range. 25, ,000 2,000,000,000

6 Long Exposure Real world 10-6 High dynamic range 10 6 Picture to 255

7 Short Exposure Real world 10-6 High dynamic range 10 6 Picture to 255

8 Image pixel (312, 284) = photons?

9 Camera Calibration Geometric How pixel coordinates relate to directions in the world Photometric How pixel values relate to radiance amounts in the world

10 Camera Calibration Geometric How pixel coordinates relate to directions in the world in other images. Photometric How pixel values relate to radiance amounts in the world in other images.

received by a given surface. This is a directional quantity.

11 The Image Acquisition Pipeline Scene radiance 2 (W/sr/m ) Lens Radiance: radiant flux emitted, reflected, transmitted or received by a given surface. This is a directional quantity. (Often called intensity confusing.) Watts per steradian per square meter

12 The Image Acquisition Pipeline Lens Shutter Scene radiance Sensor irradiance Sensor exposure 2 (W/sr/m ) 2 (W/m ) Dt Irradiance: radiant flux (power) received by a surface per unit area. Not directional (collected at surface). (Often called intensity great, also confusing.)

13 The Image Acquisition Pipeline Lens Shutter Scene radiance Sensor irradiance Sensor exposure 2 (W/sr/m ) 2 (W/m ) Dt CCD ADC Remapping Analog voltages Digital values Raw Image Pixel values JPEG Image

14 Varying Exposure

15 Camera is not a photometer! Limited dynamic range Perhaps use multiple exposures? Unknown, nonlinear response Not possible to convert pixel values to radiance Solution: Recover response curve from multiple exposures, then reconstruct the radiance map

16 Key observation: Camera is not a photometer! Scene is static, and while we might not know the absolute value of exposure at each pixel, we know that the scene radiance remains constant across the image sequence.

17 Recovering High Dynamic Range Radiance Maps from Photographs Paul Debevec Jitendra Malik Computer Science Division University of California at Berkeley August 1997

18 Ways to vary exposure Shutter Speed (*) F/stop (aperture, iris) Neutral Density (ND) Filters

19 Shutter Speed Ranges: Canon D30: 30 to 1/4,000 sec. Sony VX2000: ¼ to 1/10,000 sec. Pros: Directly varies the exposure Usually accurate and repeatable Issues: Noise in long exposures

20 The Algorithm Image series Dt = 1/64 sec Dt = 1/16 sec Dt = 1/4 sec Dt = 1 sec 1 3 Pixel Value Z = f(exposure) Exposure = Radiance Dt log Exposure = log Radiance + log Dt 2 Dt = 4 sec

21 Imaging system response function 255 Pixel value 0 log Exposure = log (Radiance Dt) (CCD photon count)

22 Pixel value The Algorithm Dt = 1/64 sec Image series Dt = 1/16 sec Dt = 1/4 sec Dt = 1 sec Pixel Value Z = f(exposure) Exposure = Radiance * Dt log Exposure = log Radiance + log Dt Dt = 4 sec Assuming unit radiance for each pixel ln Exposure

23 Pixel value Pixel value Response Curve Assuming unit radiance for each pixel After adjusting radiances to obtain a smooth response curve ln Exposure ln Exposure

24 The Math Let g(z) be the discrete inverse response function For each pixel site i in each image j, want: ln Radiance i +ln Dt j = g(z ij ) Solve the overdetermined linear system: N = # pixels P = # images N P j =1 ln Radiance i + ln Dt j g(z ij ) 2 + g (z) 2 i =1 Z max z =Z min fitting term

25 The Math Let g(z) be the discrete inverse response function For each pixel site i in each image j, want: ln Radiance i +ln Dt j = g(z ij ) Solve the overdetermined linear system: N = # pixels P = # images N P j =1 ln Radiance i + ln Dt j g(z ij ) 2 + g (z) 2 i =1 Z max z =Z min g = second derivative λ = amount of smoothing fitting term smoothness term

26 Pixel value Results: Digital Camera Kodak DCS460 1/30 to 30 sec Recovered response curve log Exposure

27 Reconstructed radiance map

28 Results: Color Film Kodak Gold ASA 100, PhotoCD

29 Recovered Response Curves Red Green Blue RGB

30 The Radiance Map

31 How do we store this?

32 Portable FloatMap (.pfm) 12 bytes per pixel, 4 for each channel sign exponent mantissa Text header similar to Jeff Poskanzer s.ppm image format: Floating Point TIFF similar PF <binary image data>

33 Radiance Format (.pic,.hdr) 32 bits / pixel Red Green Blue Exponent (145, 215, 87, 149) = (145, 215, 87) * 2^( ) = ( , , ) (145, 215, 87, 103) = (145, 215, 87) * 2^( ) = ( , , ) Ward, Greg. "Real Pixels," in Graphics Gems IV, edited by James Arvo, Academic Press, 1994

34 ILM s OpenEXR (.exr) 6 bytes per pixel, 2 for each channel, compressed 5 bit exponent 10 bit mantissa sign exponent mantissa Several lossless compression options, 2:1 typical Compatible with the half datatype in NVidia's Cg Supported natively on GeForce FX and Quadro FX Available at

35 Now What?

36 Tone Mapping How can we do this? Real World Ray Traced World (Radiance) Linear scaling?, thresholding? Suggestions? 10-6 High dynamic range 10 6 Display/ Printer to 255

37 The Radiance Map Linearly scaled to display device

38 Simple Global Operator Compression curve needs to Bring everything within range Leave dark areas alone In other words Asymptote at 255 Derivative of 1 at 0

39 Global Operator (Reinhart et al) L display L = 1+ world L world

40 Global Operator Results

41 Reinhart Operator Darkest 0.1% scaled to display device

42 What do we see? Vs.

43 Issues with multi-exposure HDR Scene and camera need to be static Camera sensors are getting better and better Display devices are fairly limited anyway (although getting better).

44 Recovering HDR radiance maps for the project Additional slides not shown in class [James Hays]

45 Debevec and Malik 1997 Image series Dt = 1/64 sec Dt = 1/16 sec Dt = 1/4 sec Dt = 1 sec Dt = 4 sec Pixel Value Z = f(exposure) Pixel Value Z = f(radiance * Dt) f -1 (Pixel Value Z) = Radiance * Dt g(pixel Value Z) = ln(radiance) + ln(dt)

46 Image Sequence Dt j = 1/64 sec 3 Dt j = 1/16 sec 3 Dt j = 1/4 sec For each pixel site i in each image j, want: g( Z ) = ln E + ln Dt Equation 2 ij g() is the unknown discrete inverse response function. Z ij is the known (but noisy) pixel value from 0 to 255 for location i in image j. E i is the unknown radiance at location i. Δt j is the known exposure time, e.g. 1/16, 1/125, etc. for image j i j in Debevec

47 Image Sequence Dt j = 1/64 sec 3 Dt j = 1/16 sec 3 Dt j = 1/4 sec g( Z ij ) ln E i = ln Dt j g( Z Dt g( Z Dt g( Z Dt 11) ln E1 = ln 12) ln E1 = ln 13) ln E1 = ln g( Z ij ) = Example of what g(z ij ) looks like 256 unknowns g( Z Dt g( Z Dt g( Z Dt 21) ln E2 = ln 22) ln E2 = ln 23) ln E2 = ln g( Z Dt 31) ln E3 = ln 32) ln E3 = ln 33) ln E3 = ln g( Z Dt g( Z Dt

48 Image Sequence Dt j = 1/64 sec 3 Dt j = 1/16 sec 3 Dt j = 1/4 sec g( Z ij ) ln E i = ln Dt j g8 ln E = ln g g 1 25 ln E1 = ln 68 ln E1 = ln Dt Dt Dt g( Z ij ) = Example of what g(z ij ) looks like 256 unknowns g g g g g g 13 ln E2 = ln 42 ln E2 = ln 117 ln E2 = ln 51 ln E3 = ln 183 ln E3 = ln 254 ln E3 = ln Dt Dt Dt Dt Dt Dt 3 2 3

49 Image Sequence Dt j = 1/64 sec 3 Dt j = 1/16 sec 3 Dt j = 1/4 sec g( Z ij ) ln E i = ln Dt j g g g ln E 1 ln E 1 ln E 1 8 = 25 = 68 = g( Z ij ) = Example of what g(z ij ) looks like 256 unknowns We expect g() to vary smoothly g g g g g g ln E 2 ln E 2 ln E 2 13 = 42 = 117 = ln E 3 ln E 3 ln E 3 51 = 183 = 254 =

50 Image Sequence Dt j = 1/64 sec 3 Dt j = 1/16 sec 3 Dt j = 1/4 sec g( Z g( Z ij ) ij ) = ln E i = ln Dt Example of what g(z ij ) looks like 256 unknowns j g g g g g g g g g ln E 1 ln E 1 ln E 1 ln E 2 ln E 2 ln E 2 ln E 3 ln E 3 ln E 3 8 = 25 = 68 = 13 = 42 = 117 = 51 = 183 = 254 = Add constraints on second derivative of g() ( g( x) g( x 1)) ( g( x + 1) g( x)) = 2 g( x) g( x 1) g( x + 1) = We expect g() to vary smoothly

51 Image Sequence Dt j = 1/64 sec 3 Dt j = 1/16 sec 3 Dt j = 1/4 sec g( Z g( Z ij ) ij ) = ln E i = ln Dt Example form of g(z ij ) 256 unknowns We expect g() to vary smoothly j g g g g g g g g g ln E 1 ln E 1 ln E 1 ln E 2 ln E 2 ln E 2 ln E 3 ln E 3 ln E 3 8 = 25 = 68 = 13 = 42 = 117 = 51 = 183 = 254 = g 2 g1 g3 = 2g 3 g 2 g 4 = 2g 254 g 253 g 255= g128= 0 num_samples * num_images equations 254 equations Lock down one point in g()

52 1 2 Image Sequence We can t trust all of these 1 values Dt j = 1/64 sec 3 Dt j = 1/16 sec 3 Dt j = 1/4 sec g( Z ij ) ln E Weight each equation by trustworthiness i = ln Dt j g g g g g g g g g ln E 1 ln E 1 ln E 1 ln E 2 ln E 2 ln E 2 ln E 3 ln E 3 ln E 3 8 = 25 = 68 = 13 = 42 = 117 = 51 = 183 = 254 = g 2 g1 g3 = 2g 3 g 2 g 4 = 2g 254 g 253 g 255= num_samples * num_images equations 254 equations g128= 0 Lock down one point in g()

53 1 2 Image Sequence We can t trust all of these 1 values Dt j = 1/64 sec 3 Dt j = 1/16 sec 3 Dt j = 1/4 sec g( Z ij ) ln E Weight each equation by trustworthiness i = ln Dt j w 8g8 w8 ln E1 = w8*-4.16 w 25g25 w25ln E1 = w25*-2.77 w 68g68 w68ln E1 = w68*-1.39 w 13g13 w13ln E2 = w13*-4.16 w 42g42 w42ln E2 = w42*-2.77 w 117g117 w117ln E2 = w117*-1.39 w 51g51 w51ln E = w51* g183 w183 E3 w 254g 254 w254 E3 w w ln = 183*-2.77 w ln = 254* w2 g 2 w2g1 w2g 3 = 2w3 g 3 w3g 2 w3g 4 = 0 0 2w254 g 254 w254g 253 w254g 255 = 0 g128= 0

54 w 8g8 w8 ln E1 = w8*-4.16 w 25g25 w25ln E1 = w25*-2.77 w 68g68 w68ln E1 = w68*-1.39 w 13g13 w13ln E2 = w13*-4.16 w 42g42 w42ln E2 = w42*-2.77 w 117g117 w117ln E2 = w117*-1.39 w 51g51 w51ln E = w51* g183 w183 E3 w 254g 254 w254 E3 w w ln = 183*-2.77 w ln = 254*-1.39 num_samples * num_images equations 2w2 g 2 w2g1 w2g 3 = 2w3 g 3 w3g 2 w3g 4 = 0 0 2w254 g 254 w254g 253 w254g 255 = 254 equations g128= 0 1 equation 0 num_samples * num_images How do we convert this to Ax=b form for Matlab? A x = b non-zero entries num_samples x = A\b num_samples 1

55 High Dynamic Range as a look [

56 High-dynamic range on smartphones Apple, iphone X/XS/XR We have a brand new feature we call smart HDR Always-on bracketed exposures: - Even when you aren t taking pictures, the camera saves to a buffer with varying exposures. - Use that data to do HDR [

57 Google Pixel phones

60 But often a global operator isn t sufficient! I lose detail. WHAT ABOUT LOCAL TONE MAPPING?

61 Local tone mapping / bilateral filter Flickr user mapgoblin

63 Contributions Contrast reduction for HDR images Local tone mapping Preserves details fewer halos than naïve filtering Fast with optimized implementation of bilateral filter (otherwise sort of slow)

64 High-dynamic-range (HDR) images CG Images Multiple exposure photo [Debevec & Malik 1997] Recover response curve HDR value for each pixel HDR sensors

65 Contrast reduction Match limited contrast of the medium Preserve details Real world 10-6 High dynamic range 10 6 Picture Low contrast

66 A typical photo Sun is overexposed Foreground is underexposed

67 Gamma (g) compression X > X g Colors are washed-out Input Gamma

68 Gamma compression on intensity Colors are OK, but details (intensity high-frequency) are blurred Intensity Gamma on intensity Color

69 Quick and dirty intensity / color separation Compute the intensity (I) by averaging the color channels. Compute the chrominance channels: (R/I, G/I, B/I)

70 Gamma compression on intensity Colors are OK, but details (intensity high-frequency) are blurred Intensity Gamma on intensity Color

71 Chiu et al Reduce contrast of low-frequencies Keep high frequencies Low-freq. Reduce low frequency High-freq. Color

72 Quick low/high frequency computation Blur intensity to get low frequencies Subtract original from low frequency - =

73 The halo nightmare For strong edges Because they contain high frequency Low-freq. Reduce low frequency High-freq. Color

74 Our approach Do not blur across edges Non-linear filtering Large-scale Output Detail Color

75 Start with Gaussian filtering Here, input is a step function + noise J = f I output input

76 Start with Gaussian filtering Spatial Gaussian f = J f I output input

77 Start with Gaussian filtering Output is blurred J = f I output input

78 Gaussian filter as weighted average Weight of x depends on distance to x J (x) = x f ( x, x ) I(x ) x x x output input

79 The problem of edges Here, I(x) pollutes our estimate J(x) It is too different across the edge J (x) = x f ( x, x ) I(x ) I(x) x I(x ) output input

80 Principle of Bilateral filtering [Tomasi and Manduchi 1998] Penalty g on the intensity difference 1 J (x) = f ( x, x ) g( I( x ) I( x)) I(x ) k( x) x x I(x ) I(x) output input

81 Bilateral filtering [Tomasi and Manduchi 1998] Spatial Gaussian f 1 J (x) = f ( x, x ) g( I( x ) I( x)) I(x ) k( x) x x output input

82 Bilateral filtering [Tomasi and Manduchi 1998] Spatial Gaussian f Gaussian g on the intensity difference 1 J (x) = f ( x, x ) g( I( x ) I( x)) I(x ) k( x) x x output input

83 Normalization factor [Tomasi and Manduchi 1998] k(x)= x f ( x, x) g( I( x) I( x)) J (x) = 1 k( x) x f ( x, x ) g( I( x ) I( x)) I(x ) x output input

84 Bilateral filtering is non-linear [Tomasi and Manduchi 1998] The weights are different for each output pixel 1 J (x) = f ( x, x) g( I( x ) I( x)) I(x ) k( x) x x x output input

85 Handling uncertainty Sometimes, not enough similar pixels Happens for specular highlights Can be detected using normalization k(x) Simple fix (average with output of neighbors) Weights with high uncertainty Uncertainty

86 Contrast reduction Input HDR image Contrast too high!

87 Contrast reduction Input HDR image Intensity Color

88 Contrast reduction Input HDR image Intensity Large scale Fast Bilateral Filter Color

89 Contrast reduction Input HDR image Intensity Large scale Fast Bilateral Filter Detail Color

90 Contrast reduction Input HDR image Intensity Large scale Reduce contrast Large scale Fast Bilateral Filter Detail Color

91 Contrast reduction Input HDR image Intensity Large scale Reduce contrast Large scale Fast Bilateral Filter Detail Preserve! Detail Color

92 Contrast reduction Input HDR image Output Intensity Large scale Reduce contrast Large scale Fast Bilateral Filter Detail Preserve! Detail Color Color

93 Informal comparison Gradient-space [Fattal et al.] Bilateral [Durand et al.] Photographic [Reinhard et al.]

94 Informal comparison Gradient-space [Fattal et al.] Bilateral [Durand et al.] Photographic [Reinhard et al.]

95 Project 2 Recover high dynamic range radiance map from multiple uncalibrated images with known exposure time. Compress high dynamic range image into low dynamic range visualization using bilateral filter based decomposition. Solving linear systems! You might need to brush up.

96 What about avoiding radiance map reconstruction entirely? Exposure Fusion [Mertens, 2007] Not shown in class; for interest / project extra credit.

97 Exposure Fusion Tom Mertens 1 Jan Kautz 2 Frank Van Reeth 1 1 EDM - Hasselt University 2 University College London, UK /

98 !!!! underexposed overexposed both

99 High Dynamic Range photography Multi-exposure sequence

100 HDR Photography HDR assembly: LDRs HDR [Mann and Picard, IS&T 95] [Debevec and Malik, SIGGRAPH 97] Tone mapping: HDR LDR [Durand and Dorsey, SIGGRAPH 02] [Reinhard et al., SIGGRAPH 02] [Fattal et al., SIGGRAPH 02] [Li et al., SIGGRAPH 05] [Lischinski et al., SIGGRAPH 06] Exposure Fusion: LDRs LDR

101 Multi-exposure Response curve calibration Assembly Tone mapping bilateral gradient display High Dynamic Range photography

102 Multi-exposure Response curve calibration Assembly Tone mapping bilateral gradient display High Dynamic Range photography Multi-exposure Exposure fusion display

103 Idea: fuse multi-exposure sequence

Image Fusion Applications Extended depth-of-field

, NPAR 04] Extended DOF Exposure Fusion Manual

104 Image Fusion Applications Extended depth-of-field [Ogden et al. 85] [Agarwala et al., SIGGRAPH 04] Multi-sensor photography [Burt et al. 93] Non-photorealistic video [Raskar et al., NPAR 04] Extended DOF Exposure Fusion Manual (Photoshop) Image pyramid [Burt et al. 93] Multisensor: IR + visible

105 [Ogden et al. 85, Burt et al. 93] Pyramid Fusion

106 Pyramid Fusion multi-scale edge filter abs max reconstruct Pyramid decomposition [Ogden et al. 85, Burt et al. 93]

107 Pyramid Fusion Exposure fusion

108 X + Challenges: 1. Good weights 2. Good blending Multi-exposure Weight

109

110 well-exposedness contrast saturation

111 weight well-exposedness saturation well-exposedness weight = st_dev(r,g,b) 1 weight = f(intensity) f(i) 0 intensity 1 contrast contrast weight = h * image, h = saturation

112 well-exposedness contrast saturation

113 Exposures Weights Naïve blending Multi-resolution blending

114 Multi-resolution Blending [Burt and Adelson, ACM TOG 83]

115 Multi-Resolution Blending x

116 Multi-Resolution Blending x + reconstruct Laplacian pyramid Gaussian pyramid [Burt and Adelson, ACM TOG 83]

117 intensity weight blended Laplacian pyramid

118 Results

119

120

121 Middle exposure Exposure fusion

122 Mean image Exposure fusion

123 Well-exposedness contrast + saturation

124 Pyramid fusion [Ogden85; Burt93] Exposure fusion Single exposure

125 [Durand and Dorsey 02] [Reinhard et al. 02] [Li et al. 05] Exposure Fusion

126 Flash/No Flash Ambient Flash Exposure Fusion Also see: [Eisemann and Durand 04, Petschnigg et al. 04, Agrawal et al. 05]

127 Conclusion Simple, fast and flexible Pixel selection + blend perceptual measures multi-resolution blending Few parameters to tweak Not physically-based Future work More measures More applications Real-time GPU implementation for video contrast saturation well-exposedness

128 Acknowledgments Photos: Min Kim Jesse Levinson Jacques Joffre Amit Agrawal European Regional Development Fund Matlab code

129

130 - Additional slides -

131 Extended DOF Graphcuts [Agrawala 04] Exposure Fusion

132 IR visible Exposure fusion

133 Middle exposure Exposure fusion

134 Mean Image Exposure fusion

135

136 Maui Sunset -2 stops +2 stops Mean image Exposure fusion

137 Maui Sunset -2 stops +2 stops Middle exposure Exposure fusion

138 Mean image Middle image Exposure fusion

139 - End -

Recap of Previous Lecture

Recap of Previous Lecture Matting foreground from background Using a single known background (and a constrained foreground) Using two known backgrounds Using lots of backgrounds to capture reflection and