Stereopsis Raul Queiroz Feitosa

Transcription

1 Stereopsis Ral Qeiroz Feitosa 5/24/2017 Stereopsis 1

2 Objetie This chapter introdces the basic techniqes for a 3 dimensional scene reconstrction based on a set of projections of indiidal points on two calibrated cameras. 5/24/2017 Stereopsis 2

3 Content Introdction Reconstrction The Correspondence Problem Rectification 5/24/2017 Stereopsis 3

4 Introdction Correspondence Detection of corresponding points in a pair of stereo images. Reconstrction Based on a set of corresponding points, compte its 3D position in the world. 5/24/2017 Stereopsis 4

5 Content Introdction Reconstrction The Correspondence Problem Rectification 5/24/2017 Stereopsis 5

6 Midpoint Trianglation OP 1 + P 1 P 2 + P 2 O = OO R P 1 P 2 P R P 1 P 2 O ^ p OP 1 normalized frames OO P 2 O ^ p O 5/24/2017 Stereopsis 6

7 Midpoint Trianglation Expressing the eqation in the normalized left camera frame, yields: OP a p ^ 1 + b (p P^ 1 Rp ) P ^ 2 + P 2 O ^ c R p = OO t O + (a,b and c can be compted) OP a p ^ 1 R P 1 OO t b P(p 1 P^ 2 Rp ) ^ crp P 2 O ^ p where R is the rotation matrix of the right camera frame and t is the ector connecting the principal points, both in relation to the left camera frame. P 2 P normalized frames R ^ ^ p O 5/24/2017 Stereopsis 7

8 Midpoint Trianglation Algorithm Gien p, p, K, R, t, K, R and t 1. Compte ^ p= K -1 p ^ p =K -1 p R=RR -1 ( rotation relatie to the left camera frame) t=-rr -1 t +t ( translation relatie to the left camera frame) 5/24/2017 Stereopsis 8

9 Midpoint Trianglation Algorithm (cont.) 2. Compte a, b and c sch that ^ ^ ^ ^ a p + b (p Rp )+ c Rp =t 3. Compte C P in the left camera frame with C ^ ^ ^ P= a p + b (p Rp )/2 4. If yo wish, transform C P to the world frame, W P= R -1 ( C P t ) t t 5/24/2017 Stereopsis 9

10 Linear Reconstrction z p = M P z p = M P p M P = 0 p M P = 0 [p ] M [p ] M P = 0 It is a system with 4 independent linear eqations ([p ] e [p ] hae rank 2). It can be applied for more than 2 cameras. 5/24/2017 Stereopsis 10

11 Geometric Reconstrction R P 1 R Q P P 2 O ^ p q ^ qadros ^ q normalizados The point Q with images q e q where the rays do intercept are sch that d 2 (p,q) + d 2 (p,q ) nder q T F q =0 is minimm a non linear system. ^ p O 5/24/2017 Stereopsis 11

12 Geometric Reconstrction Algorithm geometric interpretation The noise free projections lie on a pair of corresponding epipolar lines l=f T q epipolar line left image q p θ(t) e measred points optimal points e p θ (t) q l =Fq epipolar line right image 5/24/2017 Stereopsis 12

13 Geometric Reconstrction Algorithm otline 1. Parameterize the pencil of epipolar lines in the left image by a parameter t l(t ) 2. Using the fndamental matrix F, compte the corresponding epipolar line in the right image l (t ) 3. Express the distance d 2 (p,l(t )) + d 2 (p,l (t )) explicitly as fnction of t a polynomial of degree Find the ale of t that minimizes this distance. 5. Compte q and q and from them Q. Details can be fond in Mltiple View Geometry in compter Vision, Hartley,R. and Zisserman, A., pp 315 5/24/2017 Stereopsis 13

14 Reconstrction Error Ealation on Real Images Midpoint Geometric Standard Deiation of Gassian noise From Mltiple View Geometry in compter Vision, Hartley,R. and Zisserman, A., pp 315 5/24/2017 Stereopsis 14

15 Ealation on Real Images Points are less precisely located along the rays as the rays become more parallel ncertainty regions 5/24/2017 Stereopsis 15

16 Textring 3D Models watch Stereopsis

17 Content Introdction Reconstrction The Correspondence Problem Rectification 5/24/2017 Stereopsis 17

18 The Correspondence Problem Introdction Assme the point P in space emits light with the same energy in all directions (i.e. the srface arond P is Lambertian) I 1 I O O p p P then I(p) =I (p ) (brightness constancy constraint) where p and p are the images of P in the iews I and I. The correspondence problem consists of establishing relationships between p and p, i.e. I(p) =I ( h(p) ) ( h deformation) 5/24/2017 Stereopsis 18

19 The Correspondence Problem Introdction It is difficlt to locate an accrate match for points lying in niform neighborhoods. Also difficlt is the location of points close to edges. Therefore, before we start searching for pairs of corresponding points, it is conenient to establish which points in one image hae good characteristics for an accrate match. They mst be in neighborhoods with high dynamics. 5/24/2017 Stereopsis 19

20 The Correspondence Problem Local deformation models We look for transformations h that model the deformation on a domain W(p) arond p=[ ] T. Translational model h(p) = p +Δp W(p) Δp Affine model where A R 2 2 h(p) = A p + Δp, W(p) (A,Δp) Transformation in intensity ales I(p) =I ( h(p) ) + η( h(p) ) 5/24/2017 Stereopsis 20

21 The Correspondence Problem Matching by Sm of Sqared Differences (SSD) We choose a class of transformation and look for the particlar transformation h that minimizes the effects of noise integrated oer a window W(p), i. e., hˆ arg min h ~ pw p I ~ p I h ~ p 2 If I(p)=constant, for p W(p), the same happens for I, and the norm being minimized does not depend on h!!!! (blank wall effect) 5/24/2017 Stereopsis 21

22 The Correspondence Problem Matching by Normalized Cross-Correlation (NCC) SSD is not inariant to scaling and shifts in image intensities (deiation from the Lambertian assmptions). The Normalized Cross-Correlation (NCC) NCC h W W p I p I I h p ~ ~ I I p I Ih p p W p ~ 2 ~ I 2 I I where and are the mean intensities of I and I on W(p). -1 NCC 1. If =1, I(p) and I ( h(p) ) match perfectly. 5/24/2017 Stereopsis 22

23 The Correspondence Problem Matching by NCC For the translational model h(p) = p +Δp. W(p) _ I(p) I(p)- I _ I - = 1 W(p) _ I (p+δp) I (p+δp) -I _ I - = 2 NCC= NCC is eqal to the cosine between 1 and 2 5/24/2017 Stereopsis 23

24 The Correspondence Problem Matching by NCC (example) 5/24/2017 Stereopsis 24

25 5/24/2017 Stereopsis 25 Least Sqare Matching Affine model p = h(p) = A p +Δp Let s assme that Ths =a 11 + a 12 +b 1 and =a 21 + a 22 +b ,,, b b a a a a p p p A

26 5/24/2017 Stereopsis 26 Least Sqare Matching Affine model p = h(p) = A p +Δp Matching is established if (SSD) that can be written in more compact form as d I d I I I I,,, , -, db da da g db da da g I I

27 Least Sqare Matching Affine model p = h(p) = A p +Δp Matching is established if (SSD) where I I, x, 0 0 g 0 g 0 g g 0 g 0 g x T da 11 da12 db1 da21 da22 db2 5/24/2017 Stereopsis 27

28 5/24/2017 Stereopsis 28 Least Sqare Matching Affine model p = h(p) = A p +Δp Stacking the eqation generated by each point in W(p) yields It is a oer determined system whose soltion is gien by x = W y x n n n n n n n n g g g g g g g g g g g g I I I I,,,, y W

29 Least Sqare Matching Affine model p = h(p) = A p +Δp 1. Start with, k=0 and A b1, p b2 2. k=k+1; for each (,) in W(p) apply the transformation to compte I (, ), g and g. 3. Bild the eqation (preios slide) and compte x 4. Update the parameters A k1 A k da da k 11 k Repeat steps 2 to 4 till x goes below a certain limit. da da k 12 k 22 k1, p p k db db k 1 k 2 5/24/2017 Stereopsis 29

30 Region Growing Bilding a dense correspondence left image START right image select a seed refine location new seed N seed repository good match? yes sae match no new seed W new seed S new seed E good match create new seeds (NSEW) seeds? no sae matches yes match repository END 5/24/2017 Stereopsis 30

31 Region Growing Bilding a dense correspondence map with Region Growing 1. Select a seed of corresponding points manally, and add it in To_Check_List. 2. Take the first pair, delete it in To_Check_List and add it to Checked_List. 3. Refine location in right image sing a correlation based method 4. If constrast > min_contrast AND correlation > min_correlation, Pt pair in the Match_List. Add to the To_Check_List its neighbors d pixels to the North, to the Soth, to the East and to the West, as long as they belong neither to Checked_List nor to To_Check_List. 5. Do steps 2 to 4 till To_Check_List is empty. 5/24/2017 Stereopsis 31

32 Region Growing Example left image right image watch 5/24/2017 Stereopsis 32

33 Content Introdction Reconstrction The Correspondence Problem Rectification 5/24/2017 Stereopsis 33

34 Rectification Transforms a pair of stereo images in sch a way that both images stay on a single plane parallel to the baseline, and conjgate epipolar lines are collinear and horizontal. P conjgate epipolar lines baseline O O The search for corresponding points becomes 1D. 5/24/2017 Stereopsis 34

35 Rectification Example: original images rectified images 5/24/2017 Stereopsis 35

36 Rectification Find two projectie transformations H and H ~ ~ p =H p and p = H p that applied to each image make matching points hae approximately the same -coordinate Clearly, since the new image planes are parallel to the base line, the epipoles are in the infinity. 5/24/2017 Stereopsis 36

37 Recall Projectie Geometry 2 Dimensional If a point x lies on line l x T l = 0 The intersection x of two lines l 1 and l 2 x = l 1 l 2 The line joining points x 1 and x 2 l = x 1 x 2 Note that in P 2 parallel lines intersect at infinity (a,b,c) (a,b,c') = ( b(c'-c), a(c-c'), 0) 5/24/2017 Stereopsis 37

38 lines points Recall Projectie Geometry 2 Dimensional P 2 R 2 x =(kx 1,kx 2,k) T =(x 1,x 2,1) T for k 0 x =(x 1,x 2 ) T x =(kx 1,kx 2,0) T =(x 1,x 2,0) T for k 0 - points at infinity l =(kl 1,kl 2,k) T =(l 1,l 2,1) T for k 0 l =(a,b,c) T l =(0,0,k) T for k 0 - lines at infinity 5/24/2017 Stereopsis 38

39 Mapping epipole to infinity Assme that 0 =0 and 0 =0 (the origin is in the image center) epipole e = [ f, 0, 1] T lies on the -axis. So, the transformation G = f 0 1 takes the epipole to the point at infinity [ f, 0, 0] T 5/24/2017 Stereopsis 39

40 Mapping epipole to infinit For an arbitrarily placed point of interest p 0 and epipole e, the mapping H that makes epipolar lines rn parallel to the axis is H=GRT, where T is a transformation that moes p 0 to the origin, R is a rotation abot the origin taking the epipole to a point on the axis, and G = f 0 1 5/24/2017 Stereopsis 40

41 The matching transformation Let p and p be a pair of corresponding points. Recall that F p represents the epipolar line on the left image indced by point p on the right image. After the left image is ressampled by H, the new epipolar line will be H -T F p (why?) The mapping matrix H to be applied to the right image mst be sch that horizontal epipolar line on the right image H -T F T p =H -T F p Many matching transformations satisfy this condition. horizontal epipolar line on the left image 5/24/2017 More on Calibration and Reconstrction 41

42 Algorithm otline i. Identify a set of image-to-image matches ii. Compte the fndamental matrix F and find the epipoles e and e. iii. i. Select the projectie transformation H that maps the epipole e to the point at infinite [ f, 0, 0] T Find the matching projectie transformation H that minimizes the relatie displacement of matching points along the axis (distortion) d Hp i, H p i i. Resample both images according H and H. Details can be fond in Mltiple View Geometry in compter Vision, Hartley,R. and Zisserman, A., section pp 302 5/24/2017 Stereopsis 42

43 Example Inpt Images 5/24/2017 Stereopsis 43

44 Example Rectified Images 5/24/2017 Stereopsis 44

45 Rectification Remarks: The search for corresponding points becomes 1D. It is possible to find a match for almost eery pixel dense match map. Compensates rotation oer the image plane, which improes cross correlation. Reconstrction can be done from rectified images. 5/24/2017 Stereopsis 45

46 Assignments RECONSTRUCTION: Download the MATLAB (IteractieTrianglation) program that demonstrates stereo reconstrction interactiely. Experiment with different reconstrction approaches. Rn the TextreOerlayDemo program and experiment with the 3D model. REGION GROWING + NORMALIZED CROSS CORRELATION: Download the MATLAB demo program for Region Growing. Choose a pair of stereo images and experiment with different parameter ales. Important: follow the instrctions in the read.me file before rnning the program. RECTIFICATION: Download the MATLAB demo program for ncalibrated image rectification. Jst rn the demo sing a stereo pair as inpt. Read the MATLAB docmentation and follow the examples on Uncalibrated Stereo Image Rectification. 5/24/2017 Stereopsis 46

47 Next Topic More on Calibration 5/24/2017 Stereopsis 47