Image Stitching. Computer Vision Jia-Bin Huang, Virginia Tech. Add example. Many slides from S. Seitz and D. Hoiem

Size: px

Start display at page:

Download "Image Stitching. Computer Vision Jia-Bin Huang, Virginia Tech. Add example. Many slides from S. Seitz and D. Hoiem"

Gwendoline Conley
6 years ago
Views:

1 Iage Stitcing Add exaple Coputer Vision Jia-Bin Huang, Virginia Tec Many slides fro S. Seitz and D. Hoie

2 Adinistrative stuffs HW 3 is out due :59 PM Oct 7 Please start early. Deadlines are fir. No eails requesting extensions Getting elp? *Five* free late days witout penalty Piazza Office ours No free late dates for final projects

3 Review: Caera Projection Matrix Z Y X t r r r t r r r t r r r v f u s f v u w z y x X t x K R O w i w k w j w t R 3

4 Review: Caera Calibration Metod : Use an object (calibration grid) wit known geoetry Correspond iage points to 3d points Get least squares solution (or non-linear solution) Known 2d iage coordinates Known 3d locations wu wv w X Y Z Unknown Caera Paraeters 4

5 Z Y X s sv su Known 3d locations Known 2d iage coords Unknown Caera Paraeters u uz uy ux Z Y X v vz vy vx Z Y X Hoogeneous linear syste. Solve for s entries using linear least squares v Z v Y v X v Z Y X u Z u Y u X u Z Y X v Z v v Y X v Z Y X u Z u u Y X u Z Y X n n n n n n n n n n n n n n n n n n n n [U, S, V] = svd(a); M = V(:,end); M = resape(m,[],3)';

6 Review: Calibration by vanising points VP (2D) VP (3D) Ortogonality constraints X i X j = X i = R K p i K = p i = KRX i p i K R (R )(K )p j = Constraints for p, p 2, p 3 K = f u f v f u f f v f Unknown caera paraeters f, u, v p K (K )p 2 = p K (K )p 3 = p 2 K (K )p 3 = x u x 2 u + y v y 2 v + f 2 = Eqn () x u x 3 u + y v y 3 v + f 2 = Eqn (2) x 2 u x 3 u + y 2 v y 3 v + f 2 = Eqn (3) Eqn () Eqn (2) x u x 2 x 3 + y v y 2 y 3 = Eqn (2) Eqn (3) x 3 u x x 2 + y 3 v y y 2 = Solve for u, v f = x u x 2 u y v y 2 v

7 Review: Calibration by vanising points Rotation atrix R = r r 2 r 3 Unknown caera paraeters R p i = KRX i Set directions of vanising points X =,, X 2 =,, X 3 =,, p = Kr p 2 = Kr 2 p 3 = Kr 3 r = K p r 2 = K p 2 r 3 = K p 3 Special properties of R inv(r)=r T Eac row and colun of R as unit lengt

8 Measuring eigt v z r Slide by Steve Seitz vanising line (orizon) v x v t H t R H v y b b t b r b v v Z Z r t iage cross ratio H R 8

9 Tis class: Iage Stitcing Cobine two or ore overlapping iages to ake one larger iage Add exaple Slide credit: Vaibav Vais

10 Concepts introduced/reviewed in today s lecture Caera odel Hoograpies Solving oogeneous systes of linear equations Keypoint-based alignent RANSAC Blending How te ipone stitcer works

11 Illustration Caera Center

12 Proble set-up x = K [R t] X x' = K' [R' t'] X t=t'= x'. X x f f' x'=hx were H = K' R' R - K - Typically only R and f will cange (4 paraeters), but, in general, H as 8 paraeters

13 Hoograpy Definition General ateatics: oograpy = projective linear transforation Vision (ost coon usage): oograpy = linear transforation between iage planes two Exaples Project 3D surface into frontal view Relate two views tat differ only by rotation

14 Hoograpy exaple: Iage rectification p p To unwarp (rectify) an iage solve for oograpy H given p and p : wp =Hp

15 Hoograpy exaple: Planar apping Freedo HP Coercial

16 Iage Stitcing Algorit Overview. Detect keypoints (e.g., SIFT) 2. Matc keypoints (e.g., st /2 nd NN < tres) 3. Estiate oograpy wit four atced keypoints (using RANSAC) 4. Cobine iages

17 Coputing oograpy Assue we ave four atced points: How do we copute oograpy H? Direct Linear Transforation (DLT) x Hx ' v vv uv v u u vu uu v u ' ' ' ' ' ' w w v w u x H

18 Coputing oograpy Direct Linear Transfor Apply SVD: UDV T = A = V sallest (colun of V corr. to sallest singular value) H A n n n n n n n v v v v u v u v v v v u v u u u v u u v u Matlab [U, S, V] = svd(a); = V(:, end); Explanations of SVD and solving oogeneous linear systes

19 Coputing oograpy Assue we ave four atced points: How do we copute oograpy H? Noralized DLT. Noralize coordinates for eac iage a) Translate for zero ean b) Scale so tat average distance to origin is ~sqrt(2) ~ x Tx ~ x T x Tis akes proble better beaved nuerically (see HZ p. 7-8) 2. Copute H ~ using DLT in noralized coordinates 3. Unnoralize: H T ~ HT x i Hx i

20 HZ Tutorial 99 Coputing oograpy Assue we ave atced points wit outliers: How do we copute oograpy H? Autoatic Hoograpy Estiation wit RANSAC. Coose nuber of saples N

21 HZ Tutorial 99 Coputing oograpy Assue we ave atced points wit outliers: How do we copute oograpy H? Autoatic Hoograpy Estiation wit RANSAC. Coose nuber of saples N 2. Coose 4 rando potential atces 3. Copute H using noralized DLT 4. Project points fro x to x for eac potentially atcing pair: xi Hx i 5. Count points wit projected distance < t E.g., t = 3 pixels 6. Repeat steps 2-5 N ties Coose H wit ost inliers

22 Autoatic Iage Stitcing. Copute interest points on eac iage 2. Find candidate atces 3. Estiate oograpy H using atced points and RANSAC wit noralized DLT 4. Project eac iage onto te sae surface and blend Matlab: aketfor, itransfor

23 RANSAC for Hoograpy Initial Matced Points

24 RANSAC for Hoograpy Final Matced Points

25 RANSAC for Hoograpy

26 Coosing a Projection Surface Many to coose: planar, cylindrical, sperical, cubic, etc.

27 Planar Mapping x x f f ) For red iage: pixels are already on te planar surface 2) For green iage: ap to first iage plane

28 Planar Projection Planar Potos by Russ Hewett

29 Planar Projection Planar

30 Cylindrical Mapping x x f f ) For red iage: copute, teta on cylindrical surface fro (u, v) 2) For green iage: ap to first iage plane, tan ap to cylindrical surface

31 Cylindrical Projection Cylindrical

32 Cylindrical Projection Cylindrical

33 Planar Cylindrical

34 Recognizing Panoraas Soe of following aterial fro Brown and Lowe 23 talk Brown and Lowe 23, 27

35 Recognizing Panoraas Input: N iages. Extract SIFT points, descriptors fro all iages 2. Find K-nearest neigbors for eac point (K=4) 3. For eac iage a) Select M candidate atcing iages by counting atced keypoints (=6) b) Solve oograpy H ij for eac atced iage

36 Recognizing Panoraas Input: N iages. Extract SIFT points, descriptors fro all iages 2. Find K-nearest neigbors for eac point (K=4) 3. For eac iage a) Select M candidate atcing iages by counting atced keypoints (=6) b) Solve oograpy H ij for eac atced iage c) Decide if atc is valid (n i > n f ) # inliers # keypoints in overlapping area

37 Recognizing Panoraas (cont.) (now we ave atced pairs of iages) 4. Find connected coponents

38 Finding te panoraas

39 Finding te panoraas

40 Recognizing Panoraas (cont.) (now we ave atced pairs of iages) 4. Find connected coponents 5. For eac connected coponent a) Perfor bundle adjustent to solve for rotation (θ, θ 2, θ 3 ) and focal lengt f of all caeras b) Project to a surface (plane, cylinder, or spere) c) Render wit ultiband blending

41 Bundle adjustent for stitcing Non-linear iniization of re-projection error xˆ Hx were H = K R R - K - error N i M j k dist( x, xˆ ) Solve non-linear least squares (Levenberg- Marquardt algorit) See paper for details

42 Bundle Adjustent New iages initialised wit rotation, focal lengt of best atcing iage

43 Bundle Adjustent New iages initialised wit rotation, focal lengt of best atcing iage

44 Details to ake it look good Coosing seas Blending

45 Coosing seas Easy etod Assign eac pixel to iage wit nearest center i i2 x x Iage 2 Iage

46 Coosing seas Easy etod Assign eac pixel to iage wit nearest center Create a ask: ask(y, x) = iff pixel sould coe fro i Soot boundaries (called featering ): ask_s = ifilter(ask, gausfil); Coposite iblend = i_c.*ask + i2_c.*(-ask); i i2 x x Iage 2 Iage

47 Coosing seas Better etod: dynaic progra to find sea along well-atced regions Illustration: ttp://en.wikipedia.org/wiki/file:rocester_ny.jpg

48 Gain copensation Siple gain adjustent Copute average RGB intensity of eac iage in overlapping region Noralize intensities by ratio of averages

49 Multi-band Blending Burt & Adelson 983 Blend frequency bands over range l

50 Multiband Blending wit Laplacian Pyraid At low frequencies, blend slowly At ig frequencies, blend quickly Left pyraid blend Rigt pyraid

51 Multiband blending Laplacian pyraids. Copute Laplacian pyraid of iages and ask 2. Create blended iage at eac level of pyraid 3. Reconstruct coplete iage

52 Blending coparison (IJCV 27)

53 Blending Coparison

54 Furter reading DLT algorit: HZ p. 9 (alg 4.2), p. 585 Noralization: HZ p. 7-9 (alg 4.2) RANSAC: HZ Sec 4.7, p. 23, alg 4.6 Rick Szeliski s alignent/stitcing tutorial Recognising Panoraas: Brown and Lowe, IJCV 27 (also bundle adjustent)

55 How does ipone panoraic stitcing work? Capture iages at 3 fps Stitc te central /8 of a selection of iages Select wic iages to stitc using te acceleroeter and frae-tofrae atcing Faster and avoids radial distortion tat often occurs towards corners of iages Alignent Initially, perfor cross-correlation of sall patces aided by acceleroeter to find good regions for atcing Register by atcing points (KLT tracking or RANSAC wit FAST (siilar to SIFT) points) or correlational atcing Blending Linear (or siilar) blending, using a face detector to avoid blurring face regions and coose good face sots (not blinking, etc) ttp://

56 Tings to reeber Hoograpy relates rotating caeras Recover oograpy using RANSAC and noralized DLT Bundle adjustent iniizes reprojection error for set of related iages Details to ake it look nice (e.g., blending)

57 See you on Trusday Next class: Epipolar Geoetry and Stereo Vision

Colorado School of Mines. Computer Vision. Professor William Hoff Dept of Electrical Engineering &Computer Science.

Colorado School of Mines. Computer Vision. Professor William Hoff Dept of Electrical Engineering &Computer Science. Professor Willia Hoff Dept of Electrical Engineering &Coputer Science http://inside.ines.edu/~whoff/ 1 Caera Calibration 2 Caera Calibration Needed for ost achine vision and photograetry tasks (object