Last Lecture. Bayer pattern. Focal Length F-stop Depth of Field Color Capture. Prism. Your eye. Mirror. (flipped for exposure) Film/sensor.

Transcription

1 Last Lecture Prism Mirror (flipped for exposure) Your eye Film/sensor Focal Length F-stop Depth of Field Color Capture Light from scene lens Mirror (when viewing) Bayer pattern YungYu Chuang s slide

2 Today photomatix.com

3 The Eye The human eye is a camera! Iris - colored annulus with radial muscles Pupil - the hole (aperture) whose size is controlled by the iris What s the film? photoreceptor cells (rods and cones) in the retina

4 Two types of light-sensitive receptors Rods rod-shaped highly sensitive operate at night gray-scale vision Cones cone-shaped less sensitive operate in high light color vision Stephen E. Palmer, 2002

5 . RELATIVE ABSORBANCE (%) Physiology of Color Vision Three kinds of cones: nm. 100 S M L WAVELENGTH (nm.)

6 Electromagnetic Spectrum Human Luminance Sensitivity Function

7 References Ramanath, Snyder, Bilbro, and Sander. Demosaicking Methods for Bayer Color Arrays, Journal of Electronic Imaging, 11(3), pp Rajeev Ramanath, Wesley E. Snyder, Youngjun Yoo, Mark S. Drew, Color Image Processing Pipeline in Digital Still Cameras, IEEE Signal Processing Magazine Special Issue on Color Image Processing, vol. 22, no. 1, pp ,

8 The World in an Eye Ko Nishino Shree K. Nayar Columbia University IEEE CVPR Conference June 2004, Washington DC, USA Supported by NSF

9

10 Corneal Imaging System

11 Geometric Model of the Cornea Iris Pupil Sclera Cornea R=7.6mm t b 2.18mm rl 5.5mm

12 Finding the Limbus e limbus parameters : radii r x, ry center c, c x y tilt max e g r x * Gaussian r x e I x, y ds intensity value g r y * r y e I x, y ds

13 Self-calibration: 3D Coordinates, 3D Orientation

14 How does the World Appear in an Eye?

15 Imaging Characteristics: Viewpoint Locus P Camera Pupil N V r r c X V i S V t,, r S t, rv t, i det Z J V t,, rc 0 [Burkhard and Shealy 73; Swaminathan et al. 01]

16 Viewpoint Loci camera pupil X cornea Z

17 Viewpoint Loci camera pupil X cornea Z

18 Viewpoint Loci camera pupil cornea

19 Imaging Characteristics: Field of View camera pupil V i t b, FOV 0 2 V i z t b, 1 d closed loop of limbus incident light rays

20 Resolution and Field of View gaze direction X cornea Z

21 Resolution and Field of View resolution gaze direction X FOV cornea Z

22 Resolution and Field of View resolution gaze direction 27 person s FOV 45 X cornea Z FOV

23 What does the Eye Reveal?

24

25 Environment Map from an Eye

26 What Exactly You are Looking At Eye Image: Computed Retinal Image:

27

28

29 From Two Eyes in an Image Y X Z X Epipolar Curves Reconstructed Structure (frontal and side view) Y

30 Eyes Reveal Where the person is What the person is looking at The structure of objects

31 Implications Human Affect Studies: Social Networks Security: Human Localization Advanced Interfaces: Robots, Computers Computer Graphics: Relighting [SIGGRAPH 04]

32 Questions?

33 What do we see? Vs.

34 Camera pipeline

35 Camera pipeline

36 Camera pipeline

37 Camera pipeline

38 Camera pipeline

39 Camera pipeline Z f ( E t) 8 bits

40 Real-world response functions In general, the response function is not provided by camera makers who consider it part of their proprietary product differentiation. In addition, they are beyond the standard gamma curves.

41 Camera is not a photometer Limited dynamic range Perhaps use multiple exposures? Unknown, nonlinear response Not possible to convert pixel values to radiance Solution: Recover response curve from multiple exposures, then reconstruct the radiance map

42 Varying exposure Ways to change exposure Shutter speed Aperture Neutral density filters

43 Shutter speed Note: shutter times usually obey a power series each stop is a factor of 2 ¼, 1/8, 1/15, 1/30, 1/60, 1/125, 1/250, 1/500, 1/1000 sec Usually really is: ¼, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256, 1/512, 1/1024 sec

44 HDRI capturing from multiple exposures We want to obtain the response curve

45 HDRI capturing from multiple exposures Image series t = 2 sec t = 1 sec t = 1/2 sec t = 1/4 sec t = 1/8 sec Z ij f ( Ei t j ) f 1 ( Z ) ij E i t j g( Z ) ln E ln t, where g ln f ij i j 1

46 Idea behind the math g( Z ij ) ln E i ln t j ln E ln i t j Z ij

47 Idea behind the math g( Z ij ) ln E i ln t j ln E ln i t j Z ij

48 Idea behind the math g( Z ij ) ln E i ln t j ln E ln i t j Z ij

49 Math for recovering response curve

50 Recovering response curve The solution can be only up to a scale, add a constraint Add a hat weighting function

51 Recovering response curve We want If P=11, N~25 (typically 50 is used) We prefer that selected pixels are well distributed and sampled from constant regions. They picked points by hand. It is an overdetermined system of linear equations and can be solved using SVD

52 How to optimize? 1. Set partial derivatives zero 2. min i M 1 ( a i x b i ) 2 least -square solution of a a a : 1 2 N x b b b : 1 2 N

53 Sparse linear system 256 n n p g(0) : g(255) lne 1 : : lne n Ax=b

54 Questions Will g(127)=0 always be satisfied? Why and why not? How to find the least-square solution for an over-determined system?

55 Least-square solution for a linear system Ax m n n m m n b The are often mutually incompatible. We instead find x to minimize the norm Ax b of the residual vector Ax b. If there are multiple solutions, we prefer the one with the minimal length. x

56 Least-square solution for a linear system If we perform SVD on A and rewrite it as T 1 A UΣV Σ then xˆ VΣ 1 U T b pseudo inverse is the least-square solution. n

57 Libraries for SVD Matlab GSL Boost LAPACK (recommended) ATLAS

58 Matlab code

59 Matlab code function [g,le]=gsolve(z,b,l,w) n = 256; A = zeros(size(z,1)*size(z,2)+n+1,n+size(z,1)); b = zeros(size(a,1),1); k = 1; %% Include the data-fitting equations for i=1:size(z,1) for j=1:size(z,2) wij = w(z(i,j)+1); A(k,Z(i,j)+1) = wij; A(k,n+i) = -wij; b(k,1) = wij * B(i,j); k=k+1; end end A(k,129) = 1; %% Fix the curve by setting its middle value to 0 k=k+1; for i=1:n-2 %% Include the smoothness equations A(k,i)=l*w(i+1); A(k,i+1)=-2*l*w(i+1); A(k,i+2)=l*w(i+1); k=k+1; end x = A\b; %% Solve the system using SVD g = x(1:n); le = x(n+1:size(x,1));

60 Recovered response function

61 Constructing HDR radiance map combine pixels to reduce noise and obtain a more reliable estimation

62 Varying shutter speeds

63 Reconstructed radiance map

64 What is this for? Human perception Vision/graphics applications

65 Radiance format (.pic,.hdr,.rad) 32 bits/pixel Red Green Blue Exponent (145, 215, 87, 149) = (145, 215, 87) * 2^( ) = ( , , ) (145, 215, 87, 103) = (145, 215, 87) * 2^( ) = ( , , ) Ward, Greg. "Real Pixels," in Graphics Gems IV, edited by James Arvo, Academic Press, 1994

66 Demo

67 Image alignment

68 Median Threshold Bitmap (MTB) alignment Consider only integral translations. It is enough empirically. The inputs are N grayscale images. (You can either use the green channel or convert into grayscale by Y=(54R+183G+19B)/256) MTB is a binary image formed by thresholding the input image using the median of intensities.

69 Search for the optimal offset Try all possible offsets. Gradient descent Multiscale technique log(max_offset) levels Try 9 possibilities for the top level Scale by 2 when passing down; try its 9 neighbors

70 Threshold noise ignore pixels that are close to the threshold exclusion bitmap

71 Results Success rate = 84%. 10% failure due to rotation. 3% for excessive motion and 3% for too much high-frequency content.

72 Equipment We provide 3 sets: Contact TA for checkout.

73 HDR Video

74 Assorted pixel (Single Exposure HDR)

75 Assorted pixel

76 Assorted pixel

77 Pixel with Adaptive Exposure Control light attenuator element detector element T t+1 I t controller

78 ADR Imaging with Spatial Light Modulator image detector relay lens controllable modulator object field lens imaging lens

79 ADR Camera with LCD Attenuator LCD Electronics LCD Attenuator Video Camera Imaging Lens

80