Stereo and 3D Reconstruction

Transcription

1 Steeo and 3D Reconstuction CS635 Sping 2017 Daniel G. Aliaga Depatent of Copute Science Pudue Univesity Thanks to S. CMU fo soe of the slides

2 Poble Stateent How to ceate (ealistic) 3D odels of eisting objects and scenes in the wold? Object vs. scene Shape vs. colo vs. ateial popeties Autoatic vs. anual And any oe factos

3 3D Shape Reconstuction fo Photos

4 Fundaental Appoaches Manual odeling CAD, Sketchup, 3D Studio Ma, MS Paint Point Clouds IDAR, ase, Kinect Photogaphs photogaety and eote sensing Single Photogaph Steeo Reconstuction (2 photos) Multi-view Reconstuction naow (video?) o wide baseline

5 Fundaental Appoaches Manual odeling CAD, Sketchup, 3D Studio Ma, MS Paint Point Clouds IDAR, ase, Kinect Photogaphs photogaety and eote sensing Single Photogaph Steeo Reconstuction (2 photos) Multi-view Reconstuction naow (video?) o wide baseline

6 Definitions Caea geoety (=otion) Given coesponded points on 2 views, what ae the poses of the caeas? Coespondence geoety (=coespondence) Given a point in one view, what ae the constaints of its position in anothe view? Scene geoety (=stuctue) Given coesponded points on 2 views and the caea poses, what is the 3D location of the points?

7 X, Y, Z Steeo Rig scene left iage Assue that we know f P Y b P, Z y X baseline coesponds to ight iage, PR P R R y R Using pespective pojection (defined using coodinate syste shown) What is elationship between and X?

8 X, Y, Z Steeo Rig scene left iage Assue that we know f P Y b P, Z y X baseline coesponds to ight iage, PR P R R y R Using pespective pojection (defined using coodinate syste shown) X b 2 f Z X b R 2 f Z y f y f R Y Z

9 X, Y, Z Steeo Rig scene X left iage X b 2 f Z X b R 2 f Z b R by yr Y 2 bf Z 2 R P b, y baseline R ight iage, P R y f R y R y f R R Y Z

10 Steeo: Dispaity and Depth X, Y, Z scene X left iage X b 2 f Z X b R 2 f Z b R by yr Y 2 bf Z 2 R P b, y baseline R ight iage, P R y f R y R y f R R Y Z d R is the dispaity between coesponding left and ight iage points invesely popotional to depth dispaity inceases with baseline b

11 Steeo: Ray Tiangulation scene point optical cente iage plane optical cente (Ray) Tiangulation: copute econstuction as intesection of two ays

12 Steeo: Ray Tiangulation scene point optical cente iage plane optical cente Do two lines intesect in 3D? If so, how do you copute thei intesection?

13 Steeo: Ray Tiangulation p 3 p a p p 2 p b p 1 p 4 Equations fo the intesection: Solve fo s and t, copute p: ( p1 p2 ) ( p a pb ) ( p3 p4 ) ( p a pb ) pb p1 s( p2 p1 pa p3 t( p4 p3 ) ) 0 0 s t p ( p a pb )

14 Steeo: Vegence uncetainty of scenepoint field of view of steeo one piel optical aes of the two caeas need not be paallel 1. Field of view deceases with incease in baseline and vegence 2. Accuacy inceases with incease in baseline and vegence

15 Caea Geoety scene ~ ~ R eft fae Right fae We need to tansfo left fae to ight fae includes a otation and tanslation:

16 Caea Geoety In ati notation, we can wite as: R

17 Caea Geoety In ati notation, we can wite as: y y y z z z y z R R R

18 Caea Geoety: Othonoality Constaints R T R I (a) Rows of R ae pependicula vectos (b) Each ow of R is a unit vecto NOTE: Constaints ae NON-INEAR!

19 Caea Geoety: Poble Definition scene ~ R ~ Poble: Given ~ ~ s R Find R,,..., ) t R ( subject to (nonlinea) constaints

20 Caea Geoety: Scale Abiguity ~ ~ t R ~ R 2 ~ 2 ~ R ~ 2t R Poble: sae iage coods can be geneated by doubling Patial thus, we can find Solution: fi scale by using constaint: ~ t R only up to a scale facto! ~ ~ t R t R 1 ~ ~ R (1 additional equation) ~ t R

21 Caea Geoety: How any scene points ae needed? Each scene point gives 3 equations: y y y and 6+1 additional equations fo othonoality of otation ati constaints and scale constaint. z z z z R y R R Thus, fo n scene points, we have (3n ) equations and 12 unknowns What is the iniu value fo n?

22 Caea Geoety: Solving an Ove-deteined Syste Geneally, oe than 3 points ae used to find the 12 unknowns Foulate eo fo scene point i as: e i ( R ~ t ) R ~ R Find R & t R that iniize: E N i 1 2 ei [ 1 ( R R I) 2( t T R t R 1)]

23 Caea Geoety: A inea Estiation Assue a nea coect otation is known. Then an othogonal otation ati looks like: 1 R z y 1 z y 1 Using this ati, iteatively and linealy solve fo s and t R : iitations: ( R ~ t ) ~ R 1. ignoes noality/scale (fi by e-scaling each iteation) 2. assues good initial guess whee is the 3D otation ais and its length is the aount by which to otate How any equations/scene-points ae needed? 6 unknowns, 3 equations pe scene point, so 2 points R 0

24 Coespondence scene point optical cente iage plane optical cente

25 Coespondence scene point epipola line optical cente iage plane optical cente

26 Coespondence scene point epipola line epipola line optical cente iage plane optical cente

27 Epipola Geoety scene point epipola line epipola plane epipola line optical cente iage plane optical cente Epipola Constaint: educes coespondence poble to 1D seach along conjugate epipola lines

28 Epipola Geoety scene point epipola line epipola plane epipola line optical cente iage plane optical cente Epipola Constaint: can be epessed using the fundaental ati F

29 Epipola Geoety conveging caeas

30 Epipola Geoety otion paallel with iage plane

31 Epipola Geoety Fowad otion e e

32 Epipola Geoety Coespondence educed to looking in a sall neighbohood of a line

33 Fundaental Mati How to copute the fundaental ati? 1. geoetic eplanation 2. algebaic eplanation

34 Fundaental Mati: Geoetic Ep. X l c e e c Thus, thee is a apping l point line

35 Fundaental Mati: Geoetic Ep. X l c e e c How do you ap a point to a line?

36 Fundaental Mati: Geoetic Ep. X l c e e c Idea: - We know () s ae in a plane - Define a line by its pependicula, then we can use dot poduct; e.g., l 0 o ( c') l 0

37 Fundaental Mati: Geoetic Ep. X l c e e c What is a definition of l as pependicula to the pictued epipola line? l ( e c) ( c) l e (assue all in canonical fae of the ight-side caea)

38 Coss poduct can be epessed using ati notation: Fundaental Mati: Geoetic Ep. e l z y y z y z e e e e e e e e e ] [ e l ] [

39 How do you copute? Fundaental Mati: Geoetic Ep. Use a hoogaphy (o pojective tansfoation) to ap to (Hoogaphy: aps points in a plane to anothe plane) ,, 1 H w w w y y H

40 Fundaental Mati: Geoetic Ep. X l c e e c l [ e ] H l [ e] H F [ e] H F 0 Want l 0 T Epipola Constaint

41 Fundaental Mati: Algebaic Ep. X l c e e c R tx PX PX

42 Fundaental Mati: Algebaic Ep. PX X? X ( t) P tc Why pseudoinvese? whee P is the pseudoinvese of P Since P not squae, pseudoinvese eans PP I but solved as an optiization Recall l? PP [ e ] What is in tes of? (et s assue t 0 which eans X in on the iage plane) F [ e ] P P F 0 T Epipola Constaint

43 Coespondence: Epipola Geoety scene point epipola line epipola plane epipola line optical cente iage plane optical cente Epipola constaint educes coespondence poble to 1D seach along conjugate epipola lines

44 Coespondence: Epipola Geoety scene point epipola line epipola plane epipola line optical cente iage plane optical cente Epipola constaint can be epessed as T F 0 Fundaental ati

45 Coespondence: Epipola Geoety Inteesting case: what happens if caea otion is pue tanslation? P [I 0] P [ I t] Thus the desie to do iage ectification F? e If otion paallel to -ais e F ( H I ) T 0 iplies hoizontal 1 epipola line 0

46 Coespondence: Epipola Geoety Thus fo ectified iages, coespondence is educed to looking in a sall neighbohood of a line

47 Essential Mati Siila to the fundaental ati but includes the intinsic calibation ati, thus the equation is in tes of the noalized iage coodinates, e.g.: ' T E' 0 and E K' T FK essential ati

48 Scene Geoety scene point? Caea A Caea B Caea geoety known Coespondence and epipola geoety known What is the location of the scene point (scene geoety)?

49 Scene Geoety: inea Foulation X ~ ~a ~b M a KP a ~ a M a ~ X o Poble? Assues we know ~ But what is the value fo? w ~ b y M w b T ~ X M b KP b

50 Scene Geoety: inea Foulation ~ Recall ~ MX ~ y whee sy s s ~ ' y' w' ' y' w' '/ ' y' / w w' whee and y ae the obseved pojections s w' et, thus s? Hence? s sy 11X 12Y 13Z 14 21X 22Y 23Z 24

51 Scene Geoety: inea Foulation s sy s 11X 12Y 13Z 14 21X 22Y 23Z 31X 32Y 33Z 34 Given 24 and N caeas Fo a scene point, how any unknowns? Fo a scene point, how any caea views needed? 3+N 3N 3+N In geneal, one scene point obseved in at least two views is sufficient

52 Scene Geoety: inea Foulation b a s s ZY X Z Y X s Z Y X sy Z Y X s 2

53 Scene Geoety: inea Foulation s s Z' Y X 1 0 ' ' ' ' 0 ' ' ' ' 0 ' ' ' y y ' ' ' Z Y X s Z Y X sy Z Y X s 2 Caeas M and M

54 Scene Geoety: Nonlinea Fo. Reebe Bundle Adjustent Given initial guesses, use nonlinea least squaes to efine/copute the calibation paaetes Siple but good convegence depends on accuacy of initial guess

55 Recall ij j i j i ij j i j i ij X X y X X n E ) ~ ~ ( ) ~ ~ ( 1 Goal is 0 E Fo scene geoety, ae the unknowns X ~ Scene Geoety: Nonlinea Fo.

56 Eaple Result Using dense featue-based steeo [Pollefeys99] Net: iages and featues!