An Image Based 3D Reconstruction System for Large Indoor Scenes

Transcription

1 36 5 Vol. 36, No ACTA AUTOMATICA SINICA May, ,,,..,,,,. : 1), ; 2), ; 3),.,,. DOI,,, /SP.J An Image Based 3D Reconstruction System for Large Indoor Scenes ZHANG Feng 1 SHI Li-Min 1 SUN Feng-Mei 2 HU Zhan-Yi 1 Abstract Due to the structured nature of indoor scenes, such as the perpendicularity, parallelism, collinearity, coplanarity, etc., accustomed by human being in their daily life, usually a small reconstruction error of indoor scene could bring up dramatic visual effects. As a result few practical image based large indoor reconstruction systems are reported in the literature to our knowledge. In this work, we report an automatic image-based crime-scene reconstruction system, which includes such units as image capturing platform calibration, image point and line matching and reconstruction as well as the fusion of partial results under multiple views. The main characteristics of our system are: fully automatic, no human interaction is needed; both structural and depth information is used in line matching and reconstruction to substantially increase the reconstruction quality; in the bundle adjustment, both lines and points are optimized simultaneously. Extensive experiments on real scene have validated our system. Key words 3D reconstruction, system calibration, line matching, 3D data fusion, [1 2].,.,, [3 4], [5].,,,., : Tomasi Kanade [6]. 2004, Polleyfeys [7] Manuscript received March 30, 2009; accepted May 25, 2009 (863 ) (2007AA01Z341), ( , ) Supported by National High Technology Research and Development Program of China (863 Program) (2007AA01Z341), National Natural Science Foundation of China ( , ) National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing Faculty of Sciences, North China University of Technology, Beijing ,,,,,. 2008, Polleyfeys [8],,, GPS/INS., [9].,.,. 2009, [10],,.,,,,.,. Triggs [11],.

2 Lourakis [12].,.,. : 1 ; 2 ; 3 ; 4 ; 5.,.,,,. 2 (b),, T, α.,.,,. 1, 1.,,,., 0.02,. (a) (a) Controlled planar motion (a) (a) Rotating platform (b) (b) Platform schematic description Fig System platform 2 (a),, d, θ.,,,,,,,, 2 Fig. 2 (b) (b) Axial motion Platform motion,.,,. [13],., (, 2 (a) ).

3 5 : 627 SIFT [14], : 1.., 2 (a),., ( 2 (b), ).,. 3, 1 2, 3 1, 1, 2, 3, 5, 7,.,,. 3 (, ) Fig. 3 Process of features matching (Odd numbers represent the images captured before axial motion and even numbers represent the images captured after axial motion.) 2.,.,,. [15],.,,,,,,.,,. Hartley [16], Nister [17].,.,, T. T 2., R 3, , [18 19].,,. 2 (a),, Z, X, d, : ( ) T X p = R T 0 X c + d 0 0 (1) (, R 0, d 0 0 ) T. θ,, X 2 p = R 0 R p R T 0 X 1 c + R 0 (R p (d 0 0) T (d 0 0) T ) (2) cos θ sin θ 0, R p = sin θ cos θ (2), E = R 0 E p R T 0 (3) 0 0 sin θ, E p = 0 0 cos θ, R 0 sin θ cos θ 1 0 [ = r 1 r 2 r 3 ], a = sin θ, b = cos θ 1, E = a(r 3 r T 1 r 1 r T 3 ) + b(r 3 r T 2 + r 2 r T 3 ) (4) R 0. (1) R 0, r 3 (4), R 0., R 0.,,. (5),, R 0. m min (d( x i, F x i ) + d(x i, F T x i )) (5) R 0 i=1, F, x i, x i, m.

4 628 36, F R 0. 4.,. (1), T, d,.,,,, (2).,, 7., R 0 3,, 2, 1, 1.,,,,., SIFT.,,,. H i = K (R + T ) at K 1, i = 1, 2,, n (6) d i, K, R, T, a, d i. Fig. 4 4 Detection of vertical lines 3,.,.,.. 1..,,. 4, K, l, n = K T l.,. 2..,,,. 5,,, Fig. 5 5 Matching horizontal lines, [10].,,,.,,. 3..,,.. 6,,. (2) R c = R 0 R p R0 T, T c = R 0 (R p (d 0 0) T (d 0 0) T ), (1), X R 0 r 1, H i = K ( R c + T cr T 1 d i ) K 1, i = 1, 2,, n (7)

5 5 : 629 Fig. 6 6 Matching vertical lines,,.,,,. 4, 4,,.,, 3,. 3., 7,, { a 1 x + a 2 y + a 3 z + s = 0 (8) n 1 x + n 2 y + n 3 z = 0, a = [a 1 a 2 a 3 ] T, s, n = [n 1 n 2 n 3 ] T. [s α β] T..,,.,.,,,,.,,. 5,.,. 1. 8,, 18, 9, 10, ; 9 ; 10 ( ); 11 ; 12 ; 13 ; 14. (a) (9 3 ) (a) Three of nine images captured before axial motion (b) (9 3 ) (b) Three of nine images captured after axial motion 8 Fig. 8 Indoor scene images Fig. 7 7 Parameterization of the horizontal line in three-dimensional space n, n 1 = sin α sin β, n 2 = sin α cos β, n 3 = cos α, (a) (a) Side view of reconstructed 3D points

6 (b) (b) Vertical view of reconstructed 3D points 9 Fig. 9 Reconstructed 3D points cloud (b) (b) Vertical view of reconstructed vertical lines 12 Fig. 12 Results of reconstructed vertical lines (a) (a) Side view of reconstructed lines 10 Fig. 10 The detected images of vertical lines in three-dimensional space (a) (a) Side view of reconstructed horizontal lines (b) (b) Vertical view of reconstructed lines 13 Fig. 13 Results of reconstructed lines (b) (b) Vertical view of reconstructed horizontal lines 11 Fig. 11 Results of reconstructed horizontal lines (a) (a) Side view of reconstructed points and lines (a) (a) Side view of reconstructed vertical lines (b) (b) Vertical view of reconstructed points and lines 14 Fig. 14 Fusion of reconstructed points and lines

7 5 : ,, 24, 12, 10, ,, ,,. 17,,,..,,, ,,. (a) (12 3 ) (a) Three of twelve images captured before axial motion Fig The first image for measurement (b) (12 3 ) (b)) Three of twelve images captured after axial motion 15 Fig. 15 Indoor scene images (a) (a) Side view of reconstructed points and lines Fig The second image for measurement (b) (b)) Vertical view of reconstructed points and lines 16 Fig. 16 Fusion of reconstructed points and lines Fig The third image for measurement

8 ,,,. Table 1 1 Ratio of distances between adjacent corners on calibration board Table 2 2 Results of the measurement ( ) ( ) ( ) ( ) (%) Table 3 3 Results of the measurement ( ) ( ) ( ) ( ) (%) , 5 %,,.,. 6,,,,.,,..,,. References 1 Sequeira V, Goncalves J G M, Ribeiro M I. 3D reconstruction of indoor environments. In: Proceedings of International Conference on Image Processing. Lausanne, Switzerland: IEEE, Wang R, Luebke D. Efficient reconstruction of indoor scenes with color. In: Proceedings of the 4th International Conference on 3D Digital Imaging and Modeling. Banff, Canada: IEEE, Debevec P E, Taylor C J, Malik J. Modeling and rendering architecture from photographs: a hybrid geometry-andimaged-based approach. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. New Orleans, USA: ACM, Gibson S, Howard T. Interactive reconstruction of virtual environments from photographs, with application to sceneof-crime analysis. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology. Seoul, Korea: ACM, Hu B, Brown C. Interactive indoor scene reconstruction from image mosaics using cuboid structure. In: Proceedings of the Workshop on Motion and Video Computing. Orlando, USA: IEEE, Tomasi C, Kanade T. Shape and motion from image streams under orthography: a factorization approach. International Journal of Computer Vision, 1992, 9(2): Polleyfeys M, Gool L V, Vergauwen M, Verbiest F, Cornelis K, Tops J. Visual modeling with a hand-held camera. International Journal of Computer Vision, 2004, 59(3): Polleyfeys M, Nister D, Frahm J M, Akbarzadeh A, Mordohai P, Clipp B. Detailed real-time urban 3D reconstruction from video. International Journal of Computer Vision, 2008, 78(2-3): Shi J, Tomasi C. Good features to track. In: Proceedings of Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, Wang Z H, Wu F C, Hu Z Y. MSLD: a robust descriptor for line matching. Pattern Recognition, 2009, 42(5): Triggs B, Mclauchlan P F, Hartley R, Fitzgibbon A W. Bundle adjustment a modern synthesis. In: Proceedings of the International Workshop on Vision Algorithms: Theory and Practice. London, UK: Springer, Lourakis M, Argyros A. The Design and Implementation of a Generic Sparse Bundle Adjustment Software Package Based on the Levenberg-Marquardt Algorithm, Technical Report 340, Institute of Computer Science-FORTH, Greece, Zhang Z Y. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11):

9 5 : Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): Lhuillier M, Quan L. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(3): Hartley R. In defense of the eight-point algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(6): Nister D. An efficient solution to the five-point relative pose problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(6): Hartley R, Zisserman A. Multiple View Geometry in Computer Vision. UK: Cambridge University Press, Fitzgibbon A W, Cross G, Zisserman A. Automatic 3D model construction for turn-table sequences. In: Proceedings of the European Workshop on 3D Structure from Multiple Images of Large-Scale Environments. Freiburg, Germany: Springer, whyzf2005@163.com (ZHANG Feng Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. He received his bachelor degree from Qufu Normal University in 2002 and his master degree from Dalian University of Technology in His research interest covers 3D reconstruction and virtual reality. Corresponding author of this paper.).. lmshi@nlpr.ia.ac.cn (SHI Li-Min Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. His research interest covers image processing and 3D reconstruction.).. fmsun@163.com (SUN Feng-Mei Associate professor at the Faculty of Sciences, North China University of Technology. Her research interest covers optics and intelligent signal processing.).. huzy@nlpr.ia.ac.cn (HU Zhan-Yi Professor at the Institute of Automation, Chinese Academy of Sciences. His research interest covers camera calibration, 3D reconstruction, Hough transform, and vision guided robot navigation.)