ATIP A Tool for 3D Navigation inside a Single Image with Automatic Camera Calibration

Transcription

1 ATIP A Tool for 3D Navigation inside a Single Image with Automatic Camera Calibration Kévin Boulanger, Kadi Bouatouch, Sumanta Pattanaik IRISA, Université de Rennes I, France University of Central Florida, USA Eurographics UK Theory and Practice of Computer Graphics 2006

2 Purpose Single image Simple 3D navigation 2

3 Software architecture ATIP Maker : processes the input image, creates an.atip file ATIP Navigator : reads the.atip file, allows multi-platform 3D navigation Output Input image.atip file ATIP ATIP maker maker ATIP ATIP navigator navigator 3

4 Tour Into the Picture Original method by Horry et al. Given an input image, the perspective effect is determined by manually (mouse click) choosing a point in the image as a vanishing point The scene is manually approximated by a box by dragging its corner handles Textures are computed for each box s face from the input image The textured box is rendered from another viewpoint box with handles vanishing point 4

5 Tour Into the Picture Manual fitting of the box long and tedious Limitations supports only images with a single vanishing point horizontal and vertical lines must be parallel to the image borders careful capture of the input image is necessary (say: using a tripod) 5

6 Automatic Tour Into the Picture Contributions of ATIP: minimal user interaction more accurate calibration of the camera using multiple vanishing points input image from any orientation of the camera is acceptable the camera can be hand-held 6

7 Vanishing points Vanishing points help to calibrate the camera Always three vanishing points in an image Three combinations of finite and infinite vanishing points TIP manages only one finite vanishing point ATIP manages every situation, even with rotation around the view direction (very common without tripod) Horizontal IVP FVP Vertical IVP Vertical IVP FVP FVP FVP FVP FVP 1 finite vanishing point, 2 infinite vanishing points Horizon line 2 finite vanishing points, 1 infinite vanishing point 3 finite vanishing points 7

8 ATIP Maker pipeline First part: image processing Input image Edge detection Dominant lines detection Vanishing points estimation Three vanishing points 8

9 ATIP Maker pipeline Second part: determining parameters for TIP (camera, box, textures) Three vanishing points Camera parameter extraction Box size adjustment Texture computation Textures Geometry Input image 9

10 Edge detection Conversion to grayscale of the input image Logical AND between gradient-based and laplacian-based methods Input image Grayscale image Edges 10

11 Dominant lines detection Dominant lines: longest alignments of points Vanishing lines = subset of the set of dominant lines How: 1-to-m Hough transform of the edge points high-pass filter of the Hough transform result detection of the maxima by thresholding transform the maxima back to lines in image space Edges Hough transform High-pass filter Dominant lines 11

12 Dominant lines detection 1-to-m Hough transform Image space y Hough space Line (with polar parametrization) ρ θ x ρ ρ θ Point Point (intersection of an infinite number of lines) y θ Curve ρ = x.cos(θ) + y.sin(θ) y x ρ Aligned points Set of curves with a common point, θ corresponding to a set of aligned points x in image space 12

13 Dominant lines detection Hough space high-pass filter Simple thresholding of the Hough transform lot of noise due to strongly textured regions of the input image High-pass filter to remove the dense regions of high values (noise) Region with points to be kept Kept points Dense region of high values to be eliminated by filtering Noise eliminated Hough transform Simple thresholding High-pass filter Final result 13

14 Vanishing points estimation Vanishing point = intersection of a subset of dominant lines Projection of each dominant line onto a hemisphere Image plane Line in image space to project Accumulation of the resulting curves on the hemisphere Projection of the 2D line θ φ x Retrieve the three maxima y Project them back to image space to get the vanishing point coordinates z 14

15 Vanishing points estimation Accumulation The intersection of curves on the hemisphere corresponds to the intersection of the corresponding lines in image space (pole) Grid of accumulators i )) Finite vanishing point (close to the image center) Lines to project Image plane Final intersection point (equator) N/4 N/2 3N/4 N-1 i y z Projection of one line Intersection point of the circles x Horizontal infinite vanishing point C θ φ (x,y,z) Vertical infinite vanishing point Strips of equal area x Non-uniform subdivision of the hemisphere y z 15

16 Vanishing points estimation Three maxima retrieval The three maxima cannot be retrieved in a single step, instead : for i = 0 to 2 { filteredcells = low_pass_filter(hemisphere_cells); maxcell = maximum_value_cell(filteredcells); vanpoints[i] = project_to_image_plane(maxcell); dominantlines[i] = dominant_lines_associated_with(vanpoints[i]); hemisphere_cells = negative_accumulation(dominantlines[i], hemisphere_cells) } Hypothesis: the three resulting vanishing points correspond to orthogonal directions With this algorithm, only dominant lines that contribute to the detection of vanishing points are considered 16

17 Camera parameter extraction From the three vanishing points, finite or infinite, extract camera parameters Focal length and rotation matrix The three vanishing point directions define the world coordinate frame Process the three combinations of vanishing points different ways 17

18 Camera parameter extraction One finite VP, two infinite VPs rv 1 finite r vanishing point I 1 and I 2 directions of the infinite vanishing points in image space O position of the camera x r, y r, z r coordinate frame of the camera (fixed) (focal length) is fixed by the user (48 by default) u r f, v r, w r world coordinate frame (columns of the rotation matrix), to be found r I I I, 0 I r OV = V V, f I I, 0 1 ( ) T 1x, 1y 1 = ( ) T 1 x, 1 y 2 = ( ) T, 2x 2 y image plane V 1 I 1 P f I 2 O y v u x w z r I1xV1 x + I u = I1 x, I1y, f 1y r I 2xV1 x + I v = I 2 x, I 2 y, f ( V V f ) T r w =, 1x, 1y V 2 y 1y V 1y T T r u = r v = r w = u r r u v r r v w r r w 18

19 Camera parameter extraction Two finite VPs, one infinite VP Two known VPs ( 1 and ), represent orthogonal directions f can be computed obtained by cross product v r V V2 OV OV 1 2 ( V ) T 1x, V1 y f ( V V f ) T =, =, 2x, 2 y image plane V 2 V 1 OV1. OV2 = 0 f > 0 P f = V1 xv1 y + V2 xv2 y f x u y O v w z r u = OV OV r r r v = w u 1 1 r w = OV OV

20 Camera parameter extraction Three finite vanishing points v r 1 2 Three finite vanishing points,, and overconstrained problem The two finite VPs method used for ( v r, v r ) 1 2, ( v r, r ) and ( v r, r ) 2 v 3 3 v 1 The dot product between the third VP direction and the cross product of the two first directions gives an error value Choose the set of vectors giving minimal error v r v r 3 20

21 Box size adjustment The only step of the algorithm that requires user interaction Simpler than TIP since the camera is calibrated, only the corners (red handles) have to be moved 21

22 Texture computation Data retrieved from the input image Each box s face is subdivided into a uniform grid, each cell corresponds to a texel Each texel center point of a texture is projected onto the input image plane Bilinear interpolation gives the final color Points falling outside the input image bounds are set to black textured quadrilateral (x,y,z) image plane (x i,y i ) O 22

23 Results 23

24 Results 24

25 Results with difficult scenes Forest scene: Robust dominant lines detection from noisy edges using high-pass filter on Hough transform Non-flat surfaces: Robust «maxima on the hemisphere» computation algorithm Hand-drawn sketch: Correct detection of dominant lines from noisy edges and imprecise intersections of vanishing lines 25

26 Thanks for your attention Questions? For more information, visit