Non-line-of-sight imaging

Transcription

1 Non-line-of-sight imaging , , Computational Photography Fall 2017, Lecture 25

2 Course announcements Homework 6 will be posted tonight. - Will be due Sunday 10th. - Almost no coding, only data capture and using existing code. - You will need high-sensitivity cameras, so use the DSLRs you have picked up. Final project report deadline moved to December 15 th. - Originally was December 11 th.

3 Overview of today s lecture The non-line-of-sight (NLOS) imaging problem. Active NLOS imaging using time-of-flight imaging. Active NLOS imaging using WiFi. Passive NLOS imaging using accidental pinholes. Passive NLOS imaging using accidental reflectors. Passive NLOS imaging using corners.

4 Slide credits Many of these slides were directly adapted from: Shree Nayar (Columbia). Fadel Adib (MIT). Katie Bouman (MIT).

5 The non-line-of-sight (NLOS) imaging problem

6 Time-of-flight (ToF) imaging

7 Looking around the corner

8 Looking around the corner

9 Active NLOS imaging using time-of-flight imaging

10 Processing a single-pixel transient VD VS light source single-pixel transient camera intensity time

11 Processing a single-pixel transient VD VS light source single-pixel transient camera intensity t 1 = 10 ps time

12 Processing a single-pixel transient VD VS l in l out light source single-pixel transient camera intensity l nlos = t 1 c - l in - l out Where in the world can we find points creating this pathlength? t 1 = 10 ps time

13 Processing a single-pixel transient VD VS l in l out light source Ellipse with foci on the virtual source and detector and pathlength l nlos single-pixel transient camera intensity l nlos = t 1 c - l in - l out t 1 = 10 ps time

14 Processing a single-pixel transient VD VS l in l out light source single-pixel transient camera Ellipse with foci on the virtual source and detector and pathlength l nlos intensity l nlos = t 2 c - l in - l out t 2 = 20 ps time

15 Processing another single-pixel transient VD VS l in l out light source single-pixel transient camera intensity What assumption are we making here? Is there any other information we are not using? t 2 = 20 ps time

16 Processing another single-pixel transient VD VS l in Photons only interact with the NLOS scene once (3-bounce paths) l out light source single-pixel transient camera intensity What assumption are we making here? Is there any other information we are not using? t 2 = 20 ps time

17 Processing another single-pixel transient VD VS l in Photons only interact with the NLOS scene once (3-bounce paths) l out light source single-pixel transient camera What assumption are we making here? intensity Is there any other information we are not using? t 2 = 20 ps We need to also account for intensity time

18 Elliptic backprojection VD VS l in l out light source single-pixel transient camera

19 Elliptic backprojection

20 Elliptic backprojection

21 Reconstructing Hidden Rooms Visible wall Invisible walls

22 Reconstructing Hidden Rooms: One plane at a time Visible wall Invisible walls

23 Reconstructing Hidden Rooms: One plane at a time Visible wall Invisible walls

24 Reconstructing Hidden Rooms: One plane at a time Visible wall Invisible walls

25 Reconstructing Hidden Rooms: One plane at a time Visible wall Invisible walls

26 Reconstructing Hidden Rooms: One plane at a time Visible wall Invisible walls

27 Reconstructing Hidden Rooms: One plane at a time Visible wall Invisible walls

28 Reconstructing Rectangular Rooms Original room Reconstructed room Average AICP error for all the walls is TBD mm (TBD %). Normalized with average room length of 1.1m

29 Reconstructing Complex Shape and Reflectance wall hidden object camera what a regular camera sees what shape our camera sees wall hidden object camera what a regular camera sees what depth our camera sees

30 Active NLOS imaging using WiFi

31 Imaging through occlusions

32 Imaging through occlusions using radio frequencies

33 Key Idea

34 Challenges Wall refection is 10,000x stronger than reflections coming from behind the wall Tracking people from their reflections

35 Wi-Vi: Small, Low-Power, Wi-Fi Eliminate the wall s reflection Track people from reflections Gesture-based interface Implemented on software radios

36 How Can We Eliminate the Wall s Reflection?

37 Idea: transmit two waves that cancel each other when they reflect off static objects but not moving objects Wall is static disappears People tend to move detectable

38 Eliminating the Wall s Reflection Receive Antenna: x αx Transmit Antennas

39 Eliminating the Wall s Reflection Receive Antenna: y = h 1 x + h 2 αx 0 x h 1 α = -h 1 / h 2 αx h 2

40 Eliminating All Static Reflections

41 Eliminating All Static Reflections x αx h 1 h 2 y = h 1 x + h 2 αx Static objects (wall, furniture, etc.) have constant channels y = h 1 x + h 2 (- h 1 / h 2 )x 0 People move, therefore their channels change y = h 1 x + h 2 (- h 1 / h 2 )x Not Zero

42 How Can We Track Using Reflections?

43 RF source Tracking Motion θ Direction of reflection Antenna Array

44 Tracking Motion Direction of motion θ At any point in time, we have a single measurement Antenna Array

45 Tracking Motion Direction of motion θ θ Direction of motion Antenna Array

46 Tracking Motion Direction of motion θ Antenna Array Direction of motion Human motion emulates antenna array θ

47

48 A Through-Wall Gesture Interface Sending Commands with Gestures Two simple gestures to represent bit 0 and bit 1 Can combine sequence of gestures to convey longer message

49 Gesture Encoding Bit 0 : step forward followed by step backward Bit 1 : step backward followed by step forward Step Forward Step Backward

50 Gesture Decoding Matched Filter Peak Detector Bit 0 Bit 1 Gesture interface that works through walls and none-line-of-sight

51 Imaging through occlusions using radio frequencies Our output Camera image Head Left Arm Chest Right Arm Our RF sensor Lower Left Arm Lower Right Arm Feet

52 Traditional Imaging RF Imaging Cannot image through occlusions like walls Walls are transparent and can image through them Form 2D images using lenses No lenses at these frequencies

53 Imaging with RF No lens at these frequencies Antenna cannot distinguish bounces from different directions Antenna

54 Imaging with RF Beamforming: Use multiple antennas to scan reflections within a specific beam Antenna Array Extend to 3D with time-of-flight measurements by repeating this at every depth

55 Coarse-to-fine Scan Larger aperture (more antennas) means finer resolution Used antennas

56 Coarse-to-fine Scan Reflectors Used antennas

57 Coarse-to-fine Scan Scanned region Used antennas

58 Coarse-to-fine Scan Scanned regions Used antennas

59 Traditional Imaging RF Imaging Cannot image through occlusions like walls Walls are transparent and can image through them Form 2D images using lenses No lenses at these frequencies Get a reflection from all points: can image all the body No reflections from most points: all reflections are specular

60 Challenge: Don t get reflections from most points in RF Output of 3D RF Scan Blobs of reflection power

61 Challenge: Don t get reflections from most points in RF At frequencies that traverse walls, human body parts are specular (pure mirror)

62 Antennas Challenge: Don t get reflections from most points in RF At frequencies that traverse walls, human body parts are specular (pure mirror) Cannot Capture Reflection At every point in time, get reflections from only a subset of body parts

63 Solution Idea: Exploit Human Motion and Aggregate over Time RF Imaging Array

64 Solution Idea: Exploit Human Motion and Aggregate over Time RF Imaging Array

65 Human Walks toward Sensor 3m 2.5m 2m Chest (Largest Convex Reflector) Use it as a pivot: for motion compensation and segmentation

66 Human Walks toward Sensor 3m 2.5m 2m Right arm Head Left arm Feet Lower Torso Chest (Largest Convex Reflector) Use it as a pivot: for motion compensation and segmentation Combine the various snapshots

67 Human Walks toward Sensor

68 Implementation Hardware 2D Antenna Array Built RF circuit 1/1,000 power of WiFi USB connection to PC Rx Tx Software Coarse-to-fine algorithm implemented in GPU to generate reflection snapshots in real-time

69 Evaluation SENSOR SETUP Sensor is Placed Behind a Wall Snapshots are Sensor RF-Capture sensor placed behind the wall 15 participants Use Kinect as baseline when needed

70 Sample Captured Figures through Walls

71 y-axis (meters) y-axis (meters) y-axis (meters) y-axis (meters) Sample Captured Figures through Walls x-axis (meters) x-axis (meters) x-axis (meters) x-axis (meters)

72 Tracking result

73 Writing in the air Kinect (in red) Median Accuracy is 2cm

74 Passive NLOS imaging using accidental pinholes

75 What does this image say about the world outside?

76 Accidental pinhole camera

77 Accidental pinhole camera projected pattern on the wall window with smaller gap upside down view outside window window is an aperture

78 Accidental pinspeck camera

79 Passive NLOS imaging using accidental reflectors

80 Ophthalmology Anatomy, physiology [Helmholtz 09; ] Computer Vision Iris recognition [Daugman 93; ] HCI Gaze as an interface [Bolt 82; ] Computer Graphics Vision-realistic Rendering [Barsky et al. 02; ]

81 Corneal Imaging System

82 Geometric Model of the Cornea Iris Pupil Sclera Cornea R=7.6mm t b 2.18mm r L eccentricity = mm

83 Self-calibration: 3D Coordinates, 3D Orientation

84 Viewpoint Loci camera pupil X cornea Z

85 Viewpoint Loci camera pupil X cornea Z

86 Viewpoint Loci camera pupil cornea

87 Resolution and Field of View gaze direction X cornea Z

88 Resolution and Field of View resolution gaze direction X FOV cornea Z

89 Resolution and Field of View resolution gaze direction 27 person s FOV 45 X cornea Z FOV

90

91 Environment Map from an Eye

92 What Exactly You are Looking At Eye Image: Computed Retinal Image:

93

94

95 Corneal Stereo System right cornea left cornea epipolar plane camera pupil epipolar curve

96 From Two Eyes in an Image Y X Z X Epipolar Curves Reconstructed Structure (frontal and side view) Y

97 Eyes Reveal Where the person is What the person is looking at The structure of objects

98 Implications Human Affect Studies: Social Networks Security: Human Localization Advanced Interfaces: Robots, Computers Computer Graphics: Relighting [SIGGRAPH 04]

99 Dynamic Illumination in a Video

100 Point Source Direction from the Eye intensity

101 Point Source Trajectory from the Eye (deg.) (deg.) (deg.) (deg.)

102 Inserting a Virtual Object

103 Sampling Appearance using Eyes

104 Sampling Appearance using Eyes

105 Computed Point Source Trajectory intensity

106 Fitting a Reflectance Model

107 3D Model Reconstruction

108 Relighting under Novel Illumination

109 VisualEyes TM with Akira Yanagawa

110 Passive NLOS imaging using corners

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131 References Basic reading: Kirmani et al., Looking around the corner using ultrafast transient imaging, ICCV 2009 and IJCV Velten et al., Recovering three-dimensional shape around a corner using ultrafast time-offlight imaging, Nature Communications the two papers showing how ToF imaging can be used for looking around the corner. Abib and Katabi, See Through Walls with Wi-Fi!, SIGCOMM Abib et al., Capturing the Human Figure Through a Wall, SIGGRAPH Asia the two papers showing that WiFi can be used to see through walls. Torralba and Freeman, Accidental Pinhole and Pinspeck Cameras, CVPR the paper discussing passive NLOS imaging using accidental pinholes. Nishimo and Nayar, Corneal Imaging System: Environment from Eyes, IJCV the paper discussing passive NLOS imaging using accidental reflectors. Bouman et al., Turning corners into cameras: Principles and Methods, ICCV the paper discussing passive NLOS imaging using corners. Additional reading: Pediredla et al., Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner, ICCP the paper on NLOS room reconstruction using ToF imaging. Nishimo and Nayar, Eyes for relighting, SIGGRAPH a follow-up paper to the paper on corneal imaging, show how similar ideas can be used for relighting and other image-based rendering tasks.