Multi-View Stereo for Static and Dynamic Scenes

Transcription

1 Multi-View Stereo for Static and Dynamic Scenes Wolfgang Burgard Jan 6, 2010 Main references Yasutaka Furukawa and Jean Ponce, Accurate, Dense and Robust Multi-View Stereopsis, 2007 C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, High-quality Video View Interpolation using a Layered Representation, 2004

2 Stereo Reconstruction - Static Scene Settings Two images (2D) of the same scene Static: Scene hasn't changed accross images Acquisition from different viewpoints Camera parameters known / estimated (Zhang) Goal Reconstruct geometry (3D) of objects in scene... 2

3 Reconstruction so far Visual hull Silhouette based: Intersection of cones from silhouette back projection Image based: Project entrance/exit interval in reference images onto viewing-rays Problem: Concave surface, Image-based: View-dependent 3

4 Epipolar Geometry Scene point Image plane Epipolar Line Depth Baseline (Length b) Optical center Approach Identify correspondences between images ( correspondence problem ) Triangulation: Rays through corresponding pixels x l and x r meet at scene point depth= b focal_length disparity with disparity=x l x r 4

5 Static Scene Approach Overview Idea: Major steps Correspondences of pixels within a local area constrain each other (Local photometric consistency) Geometry as patch set Global visibility constraints Match features sparse surface patches Expand to nearby pixels dense set of patches Filter out incorrect patches ( visibility ) Patch model -> mesh 5

6 Initial Patch Set (Match step) 1/2 Divide image into 32x32 pixel cells Extract features in each cell Blobs ( DoG operator ) Corner ( Harris operator ) Uniform coverage: 4 local maxima with strongest response of each operator Triangulation with feature pairs (f,f') => 3D points Tolerance of 2 pixels from epipolar line Consider matches (f,f') if of same type 6

7 Initial Patch Set (Match step) 2/2 Many 3D points for feature f. Optimal one? Nearest to O which is photoconsistent: Patch candidate p Center c(p): 3D point Extension of p: normal Projection into image in 5x5 (7x7) axis aligned square Optimiziation consistent Maximal patches photometric consistency Refine parameters: c(p) and n(p) p photoconsistent with 2-3 images Accept, otherwise next 3D point optical center O inconsistent patch 7

8 Photometric Consistency p photoconsistent with image I? Reference image Normalized cross corelation (NCC) Energy independent similarity measure Similarity of p in I with p in reference image NCC > treshold => photoconsistent Correspondence problem not solved pointwise but for an area Local surface area considered perspectively 8

9 Expand 1/2 Initial patches too sparse For all patches p add patches p' to neighbour cells Conditions for adding No visible patch in cell No should-be-visible patch n-adjacent in cell patch p p's neighbour Close patch centers patch p p's neighbour Similar normals 9

10 Expand 2/2 Initial parameters of new patch p' n(p') = n(p) c(p') intersection of ray through Cell (i',j') with plane of p neighbour patch p' plane of p Optimiziation Maximal photometric consistency Refines c(p') and n(p') Accept if photometric consistent in 2-3 images 10

11 Filter 1/2 Remove patches outside of real surface Caused by e.g. obstacles outlier seen in one image conflicts 2 patches seen in other images Condition to remove p: p's Photometric consistency < photometric consistency of patches hidden by p Intuition: Projected outliers visible in less images than real surface patches 11

12 Filter 2/2 Remove patches inside real surface Caused by iterative scheme ( expand ) Patch added occludes "inside patch Not visible in 2-3 images => remove p Visibility here defined via depth values outlier only visible in 1 image 12

13 Mesh from Patches 1/2 Once Match (initial patches) Expand and Filter step until convergence (dense patches) Mesh from patches Initial bounding volume mesh Move vertices by forces Remesh 13

14 Mesh from Patches 2/2 Forces moving vertices Smoothness - regularization (rigidness of mesh ) Photometric consistency Initial phase: Move towards photoconsistent patches Later phases: Create patch at vertex; Optimize patch; Photoconsistency: c(p) - c*(p) Rim consistency - pull mesh towards visual cone projected surface silhouette ~ silhouette in image 14

15 Results Computation Time: Minutes to hours 15

16 Multi-View Stereo for Dynamic Scenes Input Goal: Image sequences (video) of dynamic scene Each sequence captured scene over time Variable viewpoints in video ( using geometric data ) Additional challenges: Object Movement Object Deformation Much data ( input and output ) Scene New viewpoint between recording cameras 16

17 Viewpoint Manipulation Approaches 1/2 Geometry-less approaches Jump between still cameras No software interpolation Problem: Jumping artifacts Scene Still cameras along trajectory 17

18 Viewpoint Manipulation Approaches 1/2 Geometry-less approaches Jump between still cameras No software interpolation Problem: Jumping artifacts Scene Still cameras along trajectory 18

19 Viewpoint Manipulation Approaches 1/2 Geometry-less approaches Jump between still cameras No software interpolation Problem: Jumping artifacts Scene Still cameras along trajectory 19

20 Viewpoint Manipulation Approaches 2/2 'Light-field Rendering' for dynamic scences Problem: Requires many videos Homography Project & Blend Novel view Problem: No parallax effect, i.e. foreground and background objects move with same velocity - independent from depth 20

21 Overview Snchronized videos + Camera parameters 3D Reconstruction with Image Segmentation Matte extraction ( depth disconuity artifacts ) Offline Compression Rendering using temporal two layered compression representation Interactively 21

22 Recording Setup 8 synchronized cameras 1024x fps 30 Basic camera setup Possible extension: 2D or 360 not trivial, e.g. cameras in image 2D camera setup Zhang's algorithm to estimate camera parameters 22

23 Stereo Reconstruction 1/2 Traditional 3D Stereo Reconstruction Errors around disparity disconuities => noticable visual artifacts at intensity edges Color segmentation-based stereo algorithm Segment image Similar disparity in each segment => no artifacts 1. Smooth & reduce noise 2. Merge segments (initially: each pixel ) if average color similar 3. Split/merge too large/small segments Segments 23

24 Stereo Reconstruction 2/2 Initial disparity in each segment Photometric conditions Constant disparity Refine disparity Relax constant assumption Average across images Average between segments Smoothness in each segment 24

25 Boundary Matting 1/4 Problem: At depth disconuities: Foreground pixels contain background color background foreground Hairs from foreground object having blue color from background Pixel contain foreground and background color 25

26 Boundary Matting 2/4 Left camera image Right camera image Novel view 26

27 Boundary Matting 3/4 Matting at disparity disconuities Extract foreground and background colors + alpha Two-layered representation Main layer Main layer colors Main layer depth 27

28 Boundary Matting 4/4 Matting at disparity disconuities Extract foreground and background colors + alpha Two-layered representation Boundary layer Boundary colors Boundary depth Boundary alpha 28

29 Rendering 1/2 Steps 1.Select 2 nearest cameras 2.Render into 2 buffers separate ( 1 for each camera ) Project main & boundary layer into view Depth map => 3D mesh Remove triangles across depth disconuities from main layer; Boundary mesh instead Main mesh at depth disconuity Boundary mesh at depth disconuity 29

30 Rendering 2/2 3. Blend buffers Pixels with different depth => use frontmost Similar depth => average with camera distance to view as weight Right camera nearer => more influence of pixel colors 30

31 Results Interactive rendering 5 fps 10 fps ATI 9800 PRO If all images on GPU memory: 30 fps 31

32 Summary Static scene approach Stereo reconstuction with features Photoconsistent patches Geometry: Complete 3D patch & mesh model Dynamic scene approach Stereo reconstruction with segmented images Geometry: 2 layers: Main layer + boundary layer ( matting ) 3D mesh for rendering 32

33 Discussion 33