CS5670: Computer Vision

Transcription

1 CS5670: Computer Vision Noah Snavely Multi-view stereo

2 Announcements Project 3 ( Autostitch ) due Monday 4/17 by 11:59pm

3 Recommended Reading Szeliski Chapter 11.6 Multi-View Stereo: A Tutorial Furukawa and Hernandez,

4 Multi-view Stereo Stereo Multi-view stereo

5 Multi-view Stereo Point Grey s Bumblebee XB3 Point Grey s ProFusion 25 CMU s 3D Room

6 Multi-view Stereo

7 Multi-view Stereo Input: calibrated images from several viewpoints Output: 3D object model Figures by Carlos Hernandez

8 Fua Seitz, Dyer Narayanan, Rander, Kanade Faugeras, Keriven 1998 Furukawa et al., 2010 Hernandez, Schmitt Pons, Keriven, Faugeras Furukawa, Ponce Goesele et al. 2007

9

10 Applications

11

12

13

14

15

16

17

18 Stereo: another view error depth

19

20 Why MVS? Different points on the object s surface will be more clearly visible in some subset of cameras Could have high-res closeups of some regions Some surfaces are foreshortened from certain views

21 p Cameras 2 and 3 can more clearly see point p.

22 q Cameras 1 and 2 can more clearly see point q.

23 Why MVS? Different points on the object s surface will be more clearly visible in some subset of cameras Could have high res closeups of some regions Some surfaces are foreshortened from certain views Some points may be occluded entirely in certain views

24 1 r Camera 5 can t see point r.

25 1 s Camera 1 can t see point s.

26 Why MVS? Different points on the object s surface will be more clearly visible in some subset of cameras Could have high res closeups of some regions Some surfaces are foreshortened from certain views Some points may be occluded entirely in certain views More measurements per point can reduce error

27 1 2

28 1 2

29 1 2 Estimated points contain some error.

30 1 2 Estimated points contain some error.

31 3 Estimated points contain some error. 4

32 Additional views reduce error.

33 Choosing the stereo baseline all of these points project to the same pair of pixels width of a pixel Large Baseline Small Baseline What s the optimal baseline? Too small: large depth error Too large: difficult search problem

34 The Effect of Baseline on Depth Estimation

35 z width of a pixel pixel matching score z width of a pixel

36

37 Multibaseline Stereo Basic Approach Choose a reference view Use your favorite stereo algorithm BUT replace two-view SSD with SSSD over all baselines Limitations Only gives a depth map (not an object model ) Won t work for widely distributed views:

38 Problem: visibility Some Solutions Match only nearby photos [Narayanan 98] Use NCC instead of SSD, Ignore NCC values > threshold [Hernandez & Schmitt 03]

39 Popular matching scores SSD (Sum Squared Distance) NCC (Normalized Cross Correlation) where what advantages might NCC have?

40 Single depth map often isn t enough

41 Really want full coverage

42 Idea: Combine many depth maps Many depth maps, each with error. How can we fuse these?

43 Volumetric fusion A common world-space coordinate system.

44 Volumetric fusion A common world-space coordinate system.

45

46

47 Iso surface

48 Newcombe, et. al. ISMAR 2011.

49

50

51 There are many variations on this problem.

52 Questions?

53 Are depth maps enough?

54 Are depth maps enough?

55 Depends on your application Let s assume you want to render a scene containing your real-world object. What should you capture? Shade, et. al. SIGGRAPH 98.

56 Depends on your application The object is right in front of you and you want to look all around it. Shade, et. al. SIGGRAPH 98.

57 Depends on your application Obvious dis-occlusions if a single depth map is used. Shade, et. al. SIGGRAPH 98.

58 Dis-occlusion

59 Depends on your application Close enough that you can tell it shouldn t be flat, but dis-occlusions are minimal. Shade, et. al. SIGGRAPH 98.

60 Depends on your application Far enough away that a plane can approximate it. Shade, et. al. SIGGRAPH 98.

61 Depends on your application So far away it s effectively at infinity. Shade, et. al. SIGGRAPH 98.

62 Questions?

63 Multi-view stereo from Internet Collections [Goesele, Snavely, Curless, Hoppe, Seitz, ICCV 2007]

64 appearance variation Challenges resolution massive collections

65 Law of Large Image Collections 206 Flickr images taken by 92 photographers

66 4 best neighboring views reference view Local view selection Automatically select neighboring views for each point in the image Desiderata: good matches AND good baselines

67 4 best neighboring views reference view Local view selection Automatically select neighboring views for each point in the image Desiderata: good matches AND good baselines

68 4 best neighboring views reference view Local view selection Automatically select neighboring views for each point in the image Desiderata: good matches AND good baselines

69 Results Mt. Rushmore 160 images 60 photographers St. Peter 151 images 50 photographers Trevi Fountain 106 images 51 photographers

70 Notre Dame de Paris 653 images 313 photographers

71

72

73 129 Flickr images taken by 98 photographers

74 merged model of Venus de Milo

75 56 Flickr images taken by 8 photographers

76 merged model of Pisa Cathedral

77 Accuracy compared to laser scanned model: 90% of points within 0.25% of ground truth