Coding and Modulation in Cameras

Size: px

Start display at page:

Download "Coding and Modulation in Cameras"

Mildred Riley
6 years ago
Views:

1 Mitsubishi Electric Research Laboratories Raskar 2007 Coding and Modulation in Cameras Ramesh Raskar with Ashok Veeraraghavan, Amit Agrawal, Jack Tumblin, Ankit Mohan Mitsubishi Electric Research Labs (MERL) Cambridge, MA

2 Overview Coding in Time Coded Exposure Motion Deblurring Coded Aperture Digital Refocussing Extended depth of field Optical Heterodyning Light Field Capture 4D to 2D mapping Coding in Space

3 Motion Blurred Input Photo

4 Approximate rectified crop of photo Image Deblurred by solving a linear system. No post-processing

5 Flutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006] Ferroelectric LCD shutter in front of the lens is turned opaque or transparent in a rapid binary sequence

Coded Exposure Photography: Assisting Motion Deblurring using Fluttered Shutter Short Exposure Traditional MURA Coded Shutter Captured Single Photo

6 Coded Exposure Photography: Assisting Motion Deblurring using Fluttered Shutter Short Exposure Traditional MURA Coded Shutter Captured Single Photo Deblurred Result Image is dark and noisy Result has Banding Artifacts and some spatial frequencies are lost Decoded image is as good as image of a static scene

7 Exposure choices for capturing fast moving objects Shutter On Shutter Off Time Traditional Camera Keep Shutter open for entire exposure duration The moving object creates smear Shutter On Shutter Off Time Coded Exposure Camera Flutter shutter open and closed with a psuedo-random binary sequence within exposure duration to encode the blur Shutter On Shutter Off Short Exposure Time Keep shutter open for very short duration. Avoids blur but image is dark and suffers from noise.

10 Coded Motion Blur = Preserves high spatial frequencies = Deconvolution filter frequency response is nearly flat = Deconvolution becomes a well-posed problem 0 10 Flat Ours Log Magnitude Log Magnitude pi/5 2pi/5 3pi/5 4pi/5 pi Frequency Encoding Filter (Blurring) 10 0 Flat Ours pi/5 2pi/5 3pi/5 4pi/5 pi Frequency Decoding Filter (Deblurring)

12 Defocus Blurred Photo

13 Digital Refocusing Refocused Image on Person

14 Coded Exposure Coded Aperture Temporal 1-D 1 broadband code Spatial 2-D 2 broadband code

15 Digital Refocusing

16 Coded Aperture Camera The aperture of a 100 mm lens is modified Insert a coded mask with chosen binary pattern Rest of the camera is unmodified

20 Comparison with Traditional Camera Encoded Blur Camera Traditional Camera Captured Blurred Photo Deblurred Image

21 Comparison with Small Aperture Image Small Aperture Image Captured Blurred Image Deblurred Image

22 Digital Refocusing Captured Blurred Image

23 Digital Refocusing Refocused Image on Person

24 Can we capture more information by inserting a mask?

26 Mask Sensor Lens Mask Sensor Scanner Lens Encoded Blur Camera bellow Heterodyne Light Field Camera

27 Lenslet-based Light Field camera

28 Stanford Plenoptic Camera [Ng et al 2005] Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125μ square-sided microlenses pixels lenses = pixels per lens

30 Digital Refocusing (Lenslet array)

31 Heterodyne Light Field Camera Lens Mask Sensor Scanner Lens bellow Canon flatbed scanner as the sensor. Mask is is ~ 1 cm in front of the sensor.

32 Capturing 4D light field 2D Sensor image Zoom in showing light field encoding

35 Movie: Digital Refocusing by taking 2D Projections of 4D Light Field

36 Movie: Different Views obtained from 2D Slices of 4D light field

37 How to Capture a 4D Signal with a 2D Sensor? Mask-based Modulation Software Demodulation Main Lens Object Mask Sensor Recovered Light Field Photographic Signal (Light Field) Carrier (High Frequency) Incident Modulated Signal Reference Carrier

38 Bright Background Object Reference Plane (x-plane) Main Lens Rays from Dark foreground Object Sensor θ Rays from Bright Background Object x x X Dark Foreground Object θ v d Primal Domain Light Field.

39 Bright Background Object Reference Plane (x-plane) Main Lens Rays from Dark foreground Object Sensor θ Rays from Bright Background Object x x X Dark Foreground Object θ v d Primal Domain Light Field.

40 Bright Background Object Reference Plane (x-plane) Main Lens Sensor θ x x X Dark Foreground Object θ v d Primal Domain Light Field. Fourier Transform Fourier Domain Light Field.

41 Bright Background Object Reference Plane (x-plane) Main Lens Rays from Dark foreground Object Sensor θ Rays from Bright Background Object x x X Dark Foreground Object θ v d Primal Domain Light Field. Fourier Transform Sensor: Inverse of 1-D Slice of FFT = 1-D Projection Inverse Fourier Transform Fourier Domain Light Field.

42 Bright Background Object Reference Plane (x-plane) Main Lens Mask Sensor Incident Light Field = Multiplication with the Light Field of the mask x X Dark Foreground Object θ Fourier Transform Convolution with the Fourier transform of the Mask Fourier Domain Light Field.

43 Bright Background Object Reference Plane (x-plane) Main Lens Mask Sensor x X Dark Foreground Object θ Convolution with the Fourier transform of the mask Fourier Domain Light Field.

44 Bright Background Object Reference Plane (x-plane) Main Lens Mask Sensor x X Dark Foreground Object θ Convolution with the Fourier transform of the mask Fourier Domain Light Field.

45 Bright Background Object Reference Plane (x-plane) Main Lens Mask Sensor x X Dark Foreground Object θ Convolution with the Fourier transform of the mask Fourier Domain Light Field.

46 Bright Background Object Reference Plane (x-plane) Main Lens Mask Sensor x X Dark Foreground Object θ Convolution with the Fourier transform of the mask Fourier Domain Light Field.

47 Bright Background Object Reference Plane (x-plane) Main Lens Mask Sensor x X Dark Foreground Object θ Convolution with the Fourier transform of the mask Fourier Domain Light Field.

48 Mask Sensor Lens Mask Sensor Scanner Lens Encoded Blur Camera bellow Heterodyne Light Field Camera

49 How to Capture 4D Light Field with 2D Sensor? f θ Band-limited Light Field f θ0 f x0 Sensor Slice f x Fourier Light Field Space

50 Extra sensor bandwidth cannot capture extra dimension of the light field f θ f θ0 Extra sensor bandwidth f x0 f x

51 f θ???????????? Sensor Slice f x

52 Solution: Modulation Theorem Make spectral copies of 2D light field f θ f θ0 f x0 f x Modulation Function

53 Solution: Modulation Theorem Make spectral copies of 2D light field f θ f θ0 f x0 f x Modulation Function = Sum of Cosines Mask Tile 1/f 0

54 How to Capture a 4D Signal with a 2D Sensor? Mask-based Modulation Software Demodulation Main Lens Object Mask Sensor Recovered Light Field Photographic Signal (Light Field) Carrier (High Frequency) Incident Modulated Signal Reference Carrier

55 Scanner Lens bellow Scanner 1D sensor Mask on top of scanner Mask = 4 cosines = 9 impulses in Fourier transform = 9 angular samples

56 Sensor slice captures entire Light Field f θ f θ0 f x0 Sensor Slice f x Modulation Function Modulated Light Field

57 Computing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052 2D FFT 9*9=81 such tiles 4D Light Field 181*228*9*9 4D IFFT Reshape 2D tiles into 4D planes 181*228*9*9

58 To recover light field from sensor slice, Reshape in Fourier tiles f θ Recovered Light Field f x

59 Coded Masks For Cameras Mask Sensor Lens Mask Sensor Scanner Lens bellow Encoded Blur Camera Heterodyne Light Field Camera Capture Coded Blur Full resolution digital refocussing Coarse, broadband mask in aperture Convolution of sharp image with mask Capture Light Field Complex refocussing at reduced resolution Fine, narrowband mask close to sensor Modulation of incoming light field by mask

60 Mask Sensor Lens Mask Sensor Scanner Lens bellow Encoded Blur Camera Heterodyne Light Field Camera

61 Fourier transform of optimal 1D mask pattern is a set of 1D impulses Optimal 1D Mask is sum of cosines Optimal 2D mask: Sum of cosines in 2D Number of angular samples = Number of impulses in Fourier transform of the mask along that dimension

62 1/f 0 Zoom in of the cosine mask Mask is the sum of cosines with four harmonics. Plot shows the cosines. Since the physical mask cannot have negative values, a constant needs to be added.

63 Mask-based Heterodyning Object Main Lens Mask Sensor α = (d/v) (π/2) f θ v d f θ0 α f x0 f x Mask Modulation Function

64 Implementation Lens Mask Sensor Scanner Lens bellow Light Field Camera We use Canon flatbed scanner as the sensor and Nikkor lens in front. The mask is placed is about 1 cm in front of the sensor.

Computational Cameras: Exploiting Spatial- Angular Temporal Tradeoffs in Photography

Mitsubishi Electric Research Labs (MERL) Computational Cameras Computational Cameras: Exploiting Spatial- Angular Temporal Tradeoffs in Photography Amit Agrawal Mitsubishi Electric Research Labs (MERL)