New Frontier in Visual Communication and Networking: Multi-View Imaging. Professor Tsuhan Chen 陳祖翰

Transcription

1 New Frontier in Visual Communication and Networking: Multi-View Imaging Professor Tsuhan Chen 陳祖翰 Carnegie Mellon University Pittsburgh, USA A 10-Year Journey IEEE Multimedia Signal Processing (MMSP) Technical Committee, 1996~ IEEE MMSP Workshops Princeton 1997, Los Angeles 1998, Copenhagen 1999, Cannes 2001, St. Thomas 2002, Siena 2004, Shanghai 2005, Victoria 2006 IEEE International Conf on Multimedia (ICME) New York 2000, Tokyo 2001, Lausanne 2002, Baltimore 2003, Taipei 2004, Amsterdam 2005, Toronto 2006 IEEE Transactions on Multimedia, March 1999~ Special issues: networked multimedia 2001, multimedia database 2002, multimodal interface 2003, streaming media 2004, MPEG

2 First, let us talk about sampling Which way is it rotating? Another Example [ Beat the Devil BMW Films 2002] 2

3 Sampling Theorem ωs > 2ω Nyquist Rate M Shannon, 1949 (in communication theory) Whittaker, 1964 (in math) 2WT numbers (in Fourier series) to represent a function of duration T and highest frequency W Nyquist, 1928 Gabor, 1946 Harry Nyquist [IEEE History Center] Born in Sweden PhD, Yale, 1917 AT&T/Bell Labs, US patents Contributions Quantitative study of thermal noise Vestigial sideband (VSB) transmission Nyquist diagram; stability of feedback systems Lots of signal transmission studies 3

4 Fighting with the Nyquist Rate Super-Resolution Multiple low-res images one high-res image A reconstruction problem 4

5 How Is It Possible? Does not really beat Nyquist It has limits in practice 4 of these 16 of these 64 of these 256 of these 5

6 We can beat Nyquist if we can Reconstruct One Single Image Number of all possible images = 2 >> number of all possible face images [Baker and Kanade, Hallucinating Faces ] >> human history world population We can beat Nyquist with prior Beating Nyquist with Stereo [Sawhney et. al 2001] 6

7 Beating Nyquist with Multimodality Input Video (Low Frame Rate) Image Warping Output Video (High Frame Rate) Face/Lip- Tracking Mouth Shapes and Positions Audio Audio-to- Visual Mapping Mouth Shapes (Viseme) Temporal Smoothing [Chen, Speech-assisted interpolation 1993] What else can we do to beat Nyquist? Multiview Imaging 7

8 3D vs. Multiview [Digital Michelangelo Project, Stanford] 3D (Model-Based Rendering) Light field of Michelangelo's statue of Night Multiview (Image-Based Rendering) 7D Plenoptic Function f ( V, V, V, θ, ψ, λ, t) x y z [Adelson 91] (θ,ψ) (V x,v y y,v z z ) 8

9 Lumigraph/Lightfield [Gortler et al 96] [Levoy et al 96] p p p light ray s u Outer plane (u 0,v 0 ) (s 0,t 0 ) Inner plane t v z 4D Object Captured Images t v u s 4D Data 9

10 Concentric Mosaics 3D Cameras inward The Matrix [Shum 99] EyeVision 4D (incl. time) [Kanade 01] Before Correction After Correction Super Bowl XXXV 10

11 Summary 7D plenoptic function [Adelson 91] f ( V, V, V, θ, ψ, λ, t) 5D: Stationary and monochrome [McMillan 95] 4D: Scene inside a bounded region Lumigraph [Gortler 96] Lightfield [Levoy 96] EyeVision [Kanade 01] 3D: Viewpoint along a trajectory Concentric Mosaics [Shum 99] BulletTime [ The Matrix ] 2D: Viewpoint at a single position Panorama [Chen 95], QuickTime VR x y z Where is research? Given a set of discrete samples (complete or incomplete) from the plenoptic function, the goal of IBR is to generate a continuous representation of that function [McMillan 95] Q: How many samples and where? A: Need Nyquist Sampling Theorem for IBR A redundancy-removal/compression problem 11

12 Recall u,v-s,t Parameterization p p p light ray z(v,t) s Outer plane (s 0,t 0 ) t u (u 0,v 0 ) Inner plane v z Object f ft/z(v,t) 0 v v' 0 v t 0 v t Need multidimensional spectral analysis Spectral Analysis on t-v Plane Intensity on t-v plane Spectrum Intensity on t-v plane Spectrum 12

13 Sampling for IBR Ω v d max Ωt fωv = 0 π Optimal rendering depth determined by: d opt Ω fω = 0 t v Ω v π Ω t Ω t B π (a) dmin Ωt fωv = 0 Lambertian surface No occlusion Truncating window analysis [Chai et al, 2000] π (b) Lowpass Filter Optimal Sampling Ω v Ω v B π π π 2x more compact (a) 50% fewer samples [Zhang and Chen, 2003] Ω t π Ω t Fan Filter Same rate as rectangular (b) sampling Easier to design the filter 13

14 We can do better than Nyquist Plenoptic functions are non-stationary Non-Lambertian surfaces Occlusion Poor geometry Non-uniform sampling is preferred Active IBR Determine where to capture the images Resulting a non-uniform sampling scheme Why Non-Stationary? P Object Occlusion C1 Virtual View Non-Lambertian P Object Captured Views C2 C3 C C4 Poor Geometry C1 Virtual View Captured Views C2 C3 C C4 P Object C1 Captured Views C2 Virtual View C C4 C3 14

15 Concentric Mosaic as Example Force from the left Force from the right Force proportional to color inconsistency Color Inconsistency P Object C1 Captured Views C2 α 3 α 4 α 2 Virtual View C C3 C4 15

16 Example: Capturing Non-Uniform Uniform Example: Rendering Non-Uniform Uniform 16

17 Self-Reconfigurable Camera Array [Levoy, Stanford] [Zhang and Chen, CMU] [McMillan, MIT] Setup 17

18 Details Real-time capturing/calibration/rendering 48 webcams sensor network 2 step-motors each (translation and pan) Building the next version More mobile and wireless NASA Successfully Demonstrates Innovative Nanosatellite System 005/06/ htm 18

19 How about lighting? Beyond 7D Image-Based Relighting Rendering vs. Relighting Image-based rendering Capture the scene at multiple views Interpolate between views to achieve free-viewpoint But, all these under the same lighting condition Image-based relighting Capture the scene under multiple lighting conditions Interpolation between lighting conditions Actually, linear combination 19

20 ( p q) L, Problem Formulation Q N P Lighting plane ( p q) F m, n, M Image plane ( m n) I, Surface Reflectance Function (SRF) F m,n A system identification problem Lighting patterns should be the basis functions Most effective basis functions can be obtained by principal component analysis (PCA) Result 20

21 Result (cont.) Ultimate Photography Experiences I move the camcorder around to capture the scene, but just can t capture that immersive feeling I wish I had captured that angle/moment/object/ lighting Shoot at will, and render as wished Decouple viewing from capturing Image-based rendering and relighting Mosaicing/stitching (background) Both spatial and temporal interpolation 21

22 Afterthoughts Art Inspired by Sampling 12 x 16 LEDs, 8-bit Grayscale [Jim Campbell, Portrait of a Portrait of Harry Nyquist ] 22

23 Art Inspired by Sampling 12 x 16 LEDs, 8-bit Grayscale [Jim Campbell, Portrait of a Portrait of Claude Shannon ] Art Inspired by Sampling It is the pixels or artifacts of the information that are defocused by the screen Jim Campbell [Jim Campbell, Running, Falling ] 32 x 24 LEDs, 8-bit Red 23

24 What does this say? [ Human is the best sampler Future of Visual Communication The most compelling shapes are those near to our hearts: people s faces, a gracefully moving body, a natural scene with rustling leaves and flowing water. Evolution has tuned us to these sights. By combining vision and graphics, capturing and creating images of these scenes may soon be within reach. [Lengyel, 1998] 24

25 [Chen, SBrT 05] Forget about visual communication and signal processing!!! as we defined them traditionally The future is about Mixed real and synthetic data New structured data Multi-view, multi-sensor, etc. Unstructured data It s all about fighting with Nyquist Advanced Multimedia Processing Lab Please visit us at: 25