Computer Vision 2. SS 18 Dr. Benjamin Guthier Professur für Bildverarbeitung. Computer Vision 2 Dr. Benjamin Guthier

Transcription

1 Computer Vision 2 SS 18 Dr. Benjamin Guthier Professur für Bildverarbeitung Computer Vision 2 Dr. Benjamin Guthier

2 3. HIGH DYNAMIC RANGE Computer Vision 2 Dr. Benjamin Guthier

3 Pixel Value Content of this Chapter Recap on HDR Basics HDR Applications Estimating Camera Response Local Tone Mapping Charge Computer Vision 2 Dr. Benjamin Guthier 3 3. High Dynamic Range

4 Learning Goals After this chapter, you will be able to Explain steps of the HDR pipeline from capturing to display Give examples for the application of HDR Give a motivation for camera response estimation Explain the problem formulation for two different response function estimation techniques Explain all steps of the presented local tone mapper Computer Vision 2 Dr. Benjamin Guthier 4 3. High Dynamic Range

5 Recommended Reading E. Reinhard, et al.: High Dynamic Range Imaging: Acquisition, Display and Image-based Lighting, Morgan Kaufmann, And all the papers referenced on the slides Computer Vision 2 Dr. Benjamin Guthier 5 3. High Dynamic Range

6 RECAP ON HDR BASICS Computer Vision 2 Dr. Benjamin Guthier 6 3. High Dynamic Range

7 Motivation Brightness range in a scene may be higher than what a camera can capture Computer Vision 2 Dr. Benjamin Guthier 7 3. High Dynamic Range

8 Motivation (2) Merge multiple exposures into single HDR image Conversion from pixel values to radiance HDR image = radiance map Inverse of the capturing process Δt 0 Δt 1 Δt 2 Δt3 Scene Computer Vision 2 Dr. Benjamin Guthier 8 3. High Dynamic Range

9 Dynamic Range Overview Scene LDR Image Sequence Capturing Color Conversion Aligned Images Images in Yxy Image Registration Computer Vision 2 Dr. Benjamin Guthier 9 3. High Dynamic Range

10 Overview (2) Aligned Images Images in Yxy Image Registration HDR Stitching Focus of this chapter HDR Frame Displayable Frame Tone Mapping Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

11 Capturing Process Here: Single camera captures multiple images with varying shutter speeds Result: Sequence of images I i x, y, i = 1,, N Captured in quick succession Known shutter values Δt i in ms Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

12 Pixel Value Capturing Process (2) Capturing process Light enters the lens and falls onto the sensor Sensor accumulates light over the duration of the exposure time Charge = radiance * exposure time Charge of a pixel is converted to a pixel value Based on a non-linear response function f and quantization HDR imaging is the inverse process f 1 must be estimated Converts pixel to charge f 1 is a lookup table with 256 entries Charge Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

13 (Inverted) Capturing Equation The capturing equation Radiance (L) * exposure time (Δt) induces a charge in the pixel This charge is converted to a pixel value by f i-th exposure: I i x, y = f(l x, y Δt i ) Inverting the capturing equation (Δt i and f 1 are known) One estimated radiance map L i from each exposure I i L i x, y = f 1 I i x, y Δt i Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

14 Quality Radiance Map Estimation For each position (x, y), we get one radiance estimate L i (x, y) per exposure Weighting function w gives the quality of the estimation Calculate weighted average over all estimates L x, y = σ i w I i x, y L i x, y σ i w(i i x, y ) Pixel Value Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

15 Tone Mapping Map radiance values back to [0..255] for display Radiance range is much higher than the display s capability Maintain as much of the HDR effect as possible Obvious approach: linear scaling Scale max. and min. radiance linearly to [0..255] Does not work: a lot of the HDR effect is lost! Radiance histogram Displayable range Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

16 Occurrence Sum Histogram Adjustment Tone Mapping Calculate the cumulative radiance histogram H(b) Sum up all bins of the radiance histogram h(b i ) up to radiance b H b = h(b i )/N Where N = σ bi h(b i ) b i <b h(b i ) 1 H(b) Radiance 0 Radiance Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

17 Histogram Adjustment Tone Mapping (2) Properties of the cumulative histogram H(b) b can be any radiance value 0 H b 1 H b increases faster where h(b i ) is large H b has zero slope where h b i = 0 H(b) is the desired non-linear mapping Map from radiance L(x, y) to image pixels I TM (x, y): I TM x, y = 255 H L x, y Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

18 ESTIMATING CAMERA RESPONSE Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

19 Estimating Camera Response Function Weighting function w has little effect on HDR quality Approaches differ in how they estimate camera response f Response functions are rarely linear Artistic purposes, closer model human visual system Only industrial camera have linear response Known: Images I i x, y, i = 1,, N and shutter values Δt i Assume f is monotonically increasing (higher charge higher pixel) Sufficient condition for invertibility Goal: estimate function f that maps charge to pixels Or find f 1 that maps pixel values to charge (256 charge values) Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

20 Debevec and Malik Technique Debevec, P. E., and J. Malik. "Recovering high dynamic range radiance maps from photographs." Proc. of the conf. on Computer graphics and interactive techniques. ACM, Most images in this section are taken from the paper Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

21 Example Exposure Sequence Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

22 Example Exposure Sequence (2) Actual radiance values (false color) Linear scaling to display range (0 to 255) Tone mapped image Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

23 Problem Formulation Inverse capturing equation: f 1 I i x, y = L x, y Δt i Take logarithm on both sides and define g = ln f 1 g I i x k, y k = ln L x k, y k + ln Δt i Only use a fixed number of pixels x k, y k, k = 1,, K Estimate all 256 values of g and radiance values L x k, y k By minimizing a cost function C Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

24 Problem Formulation (2) Find the values of g and the L x k, y k that minimize: K N 2 C = g I i x k, y k ln L x k, y k ln Δt i + λ g z 2 k=1 i=1 z=1 First part satisfies (log) inverse capturing equation 254 Second part ensures smoothness Parameter λ controls amount of smoothness Approximate second derivative (curvature) by g z = g z 1 2g z + g(z + 1) Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

25 Solution Formulate minimization of cost function as linear equations g I i x k, y k ln L x k, y k ln Δt i = 0 for i = 1,, N and k = 1,, K and λg z 1 2λg z + λg z + 1 = 0 for z = 1,, K variables, N K equations Choose enough pixels to be overdetermined E.g., N = 10 images and K = 50 pixel locations Solve using SVD Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

26 Choosing Good Pixel Locations Choose suitable pixel positions (x k, y k ) to estimate g Average pixels in a window around (x k, y k ) to reduce noise Use homogeneous image areas (low variance in window) Insensitive to image registration errors Cover wide range of pixel values (dark/medium/bright) Ensures many different g I i x k, y k appear in equations E.g., for I i x k, y k = 120, g(120) is one variable to solve for Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

27 Choosing Good Pixel Locations (2) Example locations and their gray value Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

28 Mitsunaga and Nayar Technique Mitsunaga, T., and S. K. Nayar. Radiometric self calibration. Computer Vision and Pattern Recognition. IEEE, Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

29 Overview Also models the inverse response function g = f 1 directly Models it as polynomial function (not a look-up table) No logarithm this time Benefit of the technique: Still works when only approximate ratios between shutter values are known Polynomial form of the inverse response function: L x k, y k Δt i = g I i x k, y k M = m=0 c m I i x k, y k m c m : Coefficients of the polynomial (to be calculated) M: Degree of the polynomial (chosen by trial and error) Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

30 Problem Formulation Known: Approximate ratio R i,i+1 = Δt i Δt i+1 between shutter values of two consecutive images (e.g., 1 2 or 1 2 ) For each pixel location x k, y k, k = 1,, K and each pair of images i = 1,, N 1 we get: R i,i+1 = L x k, y k Δt i = σ m=0 M c m I i x k, y m k L x k, y k Δt i+1 σ M m=0 c m I i+1 x k, y k m Equation defines the ratio between output values of the polynomial for two different inputs I Τ i i+1 (x k, y k ) Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

31 Problem Formulation (2) Rearrange equation: M m=0 M c m I i x k, y k m R i,i+1 m=0 Formulate as cost function to minimize: K C = k=1 N 1 i=1 M m=0 c m I i x k, y k m R i,i+1 c m I i+1 x k, y k m = 0 M m=0 c m I i+1 x k, y k m Powers of pixel values I i x k, y k m and ratios R i,i+1 are known Coefficients c m, m = 0,, M are unknown Calculate partial derivatives w.r.t. c m, set them to zero and solve linear equation system 2 Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

32 Estimate Shutter Value Ratios Sometimes only approximate Δt i are known Some cameras do not specify exact values Refine c m and R i,i+1 iteratively: Calculate c m from the approximate R i,i+1 as shown before g is determined by its coefficients c m Update R i,i+1 = g I i x k,y k by averaging over all k g I i+1 x k,y k Repeat until output values of g do not change anymore Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

33 HDR APPLICATIONS Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

34 HDR Image Processing (Example) Synthetically blurred digital image Synthetically blurred radiance map Actual blurred photograph Pixel value 255 does not reflect true energy of a pixel Direct sunlight is still very bright after blurring Source: Debevec and Malik Recovering high dynamic range radiance maps from photographs. Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

35 Image-Based Lighting 3D rendering technique that uses HDR images as light source Used when 3D objects are rendered into a real scene E.g., Computer graphics in movies 1. Capture omni-directional HDR image ( 360 or 4π sr) 2. Map HDR image to the inside of a sphere 3. Place artificial 3D object into the sphere 4. Render 3D object using HDR image as light sources Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

36 Omni-Directional HDR Image Take exposure sequence of a mirrored ball Placed where the 3D object will be HDR image represents the scene s true brightness Source: Debevec, Paul. "Image-based lighting." ACM SIGGRAPH 2006 Courses. ACM, Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

37 Rendered 3D Object Preview of the 3D object and rendered into three different environments Using HDR images from three locations Source: Debevec, Paul. "Imagebased lighting." ACM SIGGRAPH 2006 Courses. ACM, Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

38 LOCAL TONE MAPPING Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

39 Tone Mapping Tone mapping (TM) compresses range of radiance values to displayable range Displays cannot produce same brightness as real-world scenes Goal: Same subjective appearance when looking at tone mapped image and real scene Tone mapping is also used in computer graphics Uses HDR to simulate light interacting with surfaces Sunlight reflected off a 90% reflective surface should still be pure white (and not 230) Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

40 Global vs. Local Tone Mapping Global TM uses same compression function for every pixel Strictly monotonic: higher radiance higher pixel value Important to perception: local contrasts, edges, texture Less important: absolute brightness, slow intensity changes Local TM compresses each pixel s brightness individually Enhance local contrast, disregard absolute brightness May violate monotonicity Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

41 Trees Brightness range Water Sky Global vs. Local Tone Mapping (2) Real-world Local TM Global TM Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

42 Increasing Local Contrast Increase brightness difference between a pixel and its neighborhood E.g., make a dark pixel in brighter surrounding even darker Neighborhood: Area of pixels with similar brightness Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

43 Photographic Tone Mapper Reinhard, Erik, et al. "Photographic tone reproduction for digital images." ACM transactions on graphics, 21(3), Local TM motivated by Dodging and Burning (technique to in-/decrease brightness while developing photographs from film) Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

44 Overview 1. Linear scaling of radiance values (normalization) 2. Compress non-linearly to the range from 0 to 1 Can be used as a global tone mapper Global operations Local operations 3. For each pixel, find maximum circular neighborhood with similar brightness 4. Increase pixel s contrast relative to average brightness of the neighborhood Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

45 Linear Scaling Calculate log-average luminance (w: world, d: display ) തL w = exp 1 N L w x, y (x,y)log Log luminance corresponds to perceived brightness Same as geometric mean Linear scaling of radiance so that its log-average is 0.18 L x, y = 0.18 L തL w (x, y) w 18% reflectance is middle gray. Perceived as halfway between black (0%) and white (90%) Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

46 (Global) Non-Linear Compression Compress non-linearly to the range of 0,1 L(x, y) L d x, y = 1 + L(x, y) Note: L x, y 0 Example values of L d : L(x, y) L d (x, y) ~1 11 Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

47 L d (x, y) (Global) Non-Linear Compression (2) Add normalization term to obtain global TM operator L(x, y) L(x, y) L white L d x, y = 1 + L(x, y) L white : smallest luminance that will be mapped to pure white Global tone mapper for low to medium dynamic ranges L white = Scaled world luminance L(x, y) Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

48 Maximum Circular Neighborhood Create sequence of differently smoothed radiance maps V x, y, s i = L x, y G(x, y, s i ) G(x, y, s i ): Gaussian filter with standard deviation s i E.g., s i = i for i = 0,, 8 For each (x, y) and each scale s i, i > 0, check if neighborhood of size s i is homogeneous: V x, y, s i V x, y, s i 1 < T T: a small threshold Checks if average brightness in center V x, y, s i 1 is similar to average brightness in surrounding V x, y, s i For each (x, y) pick maximum scale s max for which equation holds Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

49 Maximum Circular Neighborhood (2) s 1 s 2 Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

50 Increasing Local Contrast Change non-linear compression function for each pixel L d x, y = L(x,y) 1+L(x,y) L d x, y = L(x,y) 1+V x,y,s max s max is different for every pixel local operator V x, y, s max : average brightness of a homogeneous neighborhood of maximum size V x, y, s max high/bright L d (x, y) becomes lower/darker Increases local contrast Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

51 Results Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range

52 Notes Local tone mapping operator completely replaces both (global) non-linear compression functions Contrast is only increased within homogeneous areas, not across large brightness edges Similar behavior as bilateral filters Can also be implemented using a bilateral filter Computer Vision 2 Dr. Benjamin Guthier High Dynamic Range