Intrinsic and Extrinsic Camera Parameter Estimation with Zoomable Camera for Augmented Reality

Transcription

1 Intrinsic and Extrinsic Camera Parameter Estimation with Zoomable Camera for Augmented Reality Kazuya Okada, Takafumi Taketomi, Goshiro Yamamoto, Jun Miyazaki, Hirokazu Kato Nara Institute of Science and Technology, Japan

2 Background Marker-based AR is widely used in various applications. Layout simulation [IKEA Inc.] ARToolKit TV production Most of marker-based AR applications assumes fixed intrinsic camera parameters to achieve registration between virtual and real worlds.

3 Problem of Geometric Registration while Zooming Intrinsic camera parameters will change while zooming. Zooming The accuracy of geometric registration decreases while zooming. Conventional marker-based AR Intrinsic camera parameters (Known) Extrinsic camera parameters (Unknown) Marker-based AR with Zoomable camera Intrinsic camera parameters (Unknown) Extrinsic camera parameters (Unknown) Intrinsic and extrinsic camera parameter estimation is needed to achieve accurate geometric registration.

4 Goal Simultaneous estimation of intrinsic and extrinsic camera parameters while zooming Approach Relationship between zoom values and intrinsic camera parameter change is calibrated in the offline stage. Intrinsic and extrinsic camera parameters are estimated using the following energy terms in the online stage. 1. Reprojection errors of a fiducial marker 2. Reprojection errors of tracked natural features 3. Constraint of continuity of zoom values

5 Related Work Using 2D-3D corresponding pairs of natural features [1, 2] Camera parameters are estimated based on distances between 3D positions of natural features in an object space. The accuracy of estimated camera parameters decreases depending on specific geometric relationship between camera pose and arrangement of natural features. Using 2D corresponding pairs of natural features between successive frames [3] Relative camera parameters are estimated based on epipolar geometry. Absolute extrinsic camera parameters can not be estimated. [1] M. Bujnak et al: A general solution to the P4P problem for camera with unknown focal length, CVPR 2008 [2] M. Bujnak et al: New efficient solution to the absolute pose problem for camera with unknown focal length, ACCV 2010 [3] H. Stewenius et al: A minimal solution for relative pose with unknown focal length, CVPR 2005

6 Overview of The Proposed Method Offline stage Online stage Zoom value Calibration between zoom values and intrinsic camera parameter change Energy based on epipolar geometry Continuity of zoom value Energy based on fiducial marker Balancing between each energy term Intrinsic and extrinsic camera parameter estimation based on energy minimization

7 Camera Calibration Relationship between zoom values and intrinsic camera parameters is calibrated in the offline stage. K z z 0 0 f x, f y : focal length f x 0 z 0 u, v: center of projection f y u v z z 1 f x (z) f y (z) u(z) v(z) Intrinsic camera parameters are calibrated for each zoom value and then intrinsic camera parameters are represented using the zoom value z.

8 Definition of The Energy Function The energy function consists of three terms E = E ep + w mk E mk + w z E z E ep : Reprojection errors based on epipolar constraint E mk : Reprojection errors of fiducial marker E z : Continuity of zoom values w mk, w z : Weight for balancing each term

9 E ep : Energy Term Based on Epipolar Constraint Reprojection errors are defined based on epipolar constraint using tracked natural features. k P Reprojection error d E ep = 1 k i=1 d i p n-m p n k: number of corresponding pairs e n :epipolar line Previous frame m, p n : coordinates of natural features in p n m frame m and frame n e n-m Epipolar line e n Current frame n The reprojection error is defined as the distance between an epipolar line and a detected natural feature position in the input image.

10 Energy Term Based on Fiducial Marker Reprojection errors are calculated from correspondences between fiducial marker corners in an input image and its reprojected points. 4 E mk = K z TP i p i 2 i=1 θ The accuracy of marker-based estimation result will be unstable when the optical axis is perpendicular to a marker plane. Weight for E mk is calculated from the angle between optical axis and marker plane. w mk = 4 π 2 θ2 + ε w mk θ[degree]

11 Energy Term Based on Continuity of Zoom Values Zoom values within successive frames do not change drastically. E z = z n z n 1 2 z n : Zoom value of current frame z n 1 : Zoom value of previous frame The relationship between zoom values and intrinsic camera parameters is not proportional. Focal length is drastically changed at a large zoom value. Weight for E z depends on f x (z) w z = 1 f x (z) Zoom value z f x f y u v

12 Camera Parameter Estimation Based on Energy Minimization Camera parameters are estimated by minimizing the energy function using the Levenberg-Marquardt algorithm with the Geman-McLure M-estimator. Y X Z Initial value:z n 1 α Initial value:z n 1 Previous frame m Ground Truth:z n Initial value:t init (+, +, ) Ground Truth:t true (+, +, +) Initial value:z n 1 + α In order to avoid a local minimum problem, the optimization process is executed using three different initial values.

13 Experiments The accuracy of estimated camera parameters are compared to the previous method [Bujnak et al. ACCV2010] in simulated and real environments. Simulated Environment Virtual camera motion in this environment is acquired by using ARToolkit D points were randomly generated in the 3-D space (500mm 500mm 500mm). Corresponding pairs were obtained by projecting these 3-D points into virtual cameras. Gaussian noise was added, with mean equal to zero and standard deviation of s = 2:0. Real Environment The results of overlaying the CG object and estimated camera paths are compared to the previous method.

14 Focal length[mm] Center of projection [pixel] Result of Camera Calibration 1200 fx(z) fy(z) 400 u(z) v(z) Zoom value Zoom Value In the following experiments, we use the spline fitting result of f x (z), f y (z), u(z), and v(z).

15 Focal length[mm] Position error[mm] Focal length [mm] Position error [mm] Quantitative Evaluation in Simulated Environment Bujnk et al. ACCV fx (Estimated) Frame number Proposed method fx (Estimated) fx (Ground truth) fx (Ground truth) Frame number Frame number Frame number The accuracies of estimated camera parameters are drastically improved in the proposed method.

16 Qualitative Evaluation in Real Environment A virtual cube is overlaid to a real cube. Bujnak et al. ACCV2010 Proposed method Our method can register the virtual cube more accurate than that of the previous method.

17 Estimated Camera Path Estimated camera path (Bujnak et al. ACCV2010) Estimated camera path (Proposed method) The proposed method can estimate the camera path with more stability than that of the previous method.

18 Conclusion We proposed an intrinsic and extrinsic camera parameter estimation method for zoomable camera. Camera calibration Relationship between zoom values and the intrinsic camera parameter change is calibrated in advance Camera parameter estimation Two energy terms are added to the conventional marker-based camera parameter estimation framework. The proposed method can estimate intrinsic and extrinsic camera parameters more accurately than the state of the art method [Bujnak et al. ACCV2010]. Future work Lens distortion estimation will be incorporated into the proposed method.