36 5 Vol. 36, No. 5 2010 5 ACTA AUTOMATICA SINICA May, 2010 1 1 2 1,,,..,,,,. : 1), ; 2), ; 3),.,,. DOI,,, 10.3724/SP.J.1004.2010.00625 An Image Based 3D Reconstruction System for Large Indoor Scenes ZHANG Feng 1 SHI Li-Min 1 SUN Feng-Mei 2 HU Zhan-Yi 1 Abstract Due to the structured nature of indoor scenes, such as the perpendicularity, parallelism, collinearity, coplanarity, etc., accustomed by human being in their daily life, usually a small reconstruction error of indoor scene could bring up dramatic visual effects. As a result few practical image based large indoor reconstruction systems are reported in the literature to our knowledge. In this work, we report an automatic image-based crime-scene reconstruction system, which includes such units as image capturing platform calibration, image point and line matching and reconstruction as well as the fusion of partial results under multiple views. The main characteristics of our system are: fully automatic, no human interaction is needed; both structural and depth information is used in line matching and reconstruction to substantially increase the reconstruction quality; in the bundle adjustment, both lines and points are optimized simultaneously. Extensive experiments on real scene have validated our system. Key words 3D reconstruction, system calibration, line matching, 3D data fusion, [1 2].,.,, [3 4], [5].,,,., : Tomasi Kanade [6]. 2004, Polleyfeys [7] 2009-03-30 2009-05-25 Manuscript received March 30, 2009; accepted May 25, 2009 (863 ) (2007AA01Z341), (60835003, 60673104) Supported by National High Technology Research and Development Program of China (863 Program) (2007AA01Z341), National Natural Science Foundation of China (60835003, 60673104) 1. 100190 2. 100044 1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190 2. Faculty of Sciences, North China University of Technology, Beijing 100044,,,,,. 2008, Polleyfeys [8],,, GPS/INS., [9].,.,. 2009, [10],,.,,,,.,. Triggs [11],.
626 36 Lourakis [12].,.,. : 1 ; 2 ; 3 ; 4 ; 5.,.,,,. 2 (b),, T, α.,.,,. 1, 1.,,,., 0.02,. (a) (a) Controlled planar motion (a) (a) Rotating platform (b) (b) Platform schematic description Fig. 1 1 2 System platform 2 (a),, d, θ.,,,,,,,, 2 Fig. 2 (b) (b) Axial motion Platform motion,.,,. [13],., (, 2 (a) ).
5 : 627 SIFT [14], : 1.., 2 (a),., ( 2 (b), ).,. 3, 1 2, 3 1, 1, 2, 3, 5, 7,.,,. 3 (, ) Fig. 3 Process of features matching (Odd numbers represent the images captured before axial motion and even numbers represent the images captured after axial motion.) 2.,.,,. [15],.,,,,,,.,,. Hartley [16], Nister [17].,.,, T. T 2., R 3, 5. 3.., [18 19].,,. 2 (a),, Z, X, d, : ( ) T X p = R T 0 X c + d 0 0 (1) (, R 0, d 0 0 ) T. θ,, X 2 p = R 0 R p R T 0 X 1 c + R 0 (R p (d 0 0) T (d 0 0) T ) (2) cos θ sin θ 0, R p = sin θ cos θ 0. 0 0 1 (2), E = R 0 E p R T 0 (3) 0 0 sin θ, E p = 0 0 cos θ, R 0 sin θ cos θ 1 0 [ = r 1 r 2 r 3 ], a = sin θ, b = cos θ 1, E = a(r 3 r T 1 r 1 r T 3 ) + b(r 3 r T 2 + r 2 r T 3 ) (4) R 0. (1) R 0, r 3 (4), R 0., R 0.,,. (5),, R 0. m min (d( x i, F x i ) + d(x i, F T x i )) (5) R 0 i=1, F, x i, x i, m.
628 36, F R 0. 4.,. (1), T, d,.,,,, (2).,, 7., R 0 3,, 2, 1, 1.,,,,., SIFT.,,,. H i = K (R + T ) at K 1, i = 1, 2,, n (6) d i, K, R, T, a, d i. Fig. 4 4 Detection of vertical lines 3,.,.,.. 1..,,. 4, K, l, n = K T l.,. 2..,,,. 5,,, Fig. 5 5 Matching horizontal lines, [10].,,,.,,. 3..,,.. 6,,. (2) R c = R 0 R p R0 T, T c = R 0 (R p (d 0 0) T (d 0 0) T ), (1), X R 0 r 1, H i = K ( R c + T cr T 1 d i ) K 1, i = 1, 2,, n (7)
5 : 629 Fig. 6 6 Matching vertical lines,,.,,,. 4, 4,,.,, 3,. 3., 7,, { a 1 x + a 2 y + a 3 z + s = 0 (8) n 1 x + n 2 y + n 3 z = 0, a = [a 1 a 2 a 3 ] T, s, n = [n 1 n 2 n 3 ] T. [s α β] T..,,.,.,,,,.,,. 5,.,. 1. 8,, 18, 9, 10, 1 600 1 200. 8 ; 9 ; 10 ( ); 11 ; 12 ; 13 ; 14. (a) (9 3 ) (a) Three of nine images captured before axial motion (b) (9 3 ) (b) Three of nine images captured after axial motion 8 Fig. 8 Indoor scene images Fig. 7 7 Parameterization of the horizontal line in three-dimensional space n, n 1 = sin α sin β, n 2 = sin α cos β, n 3 = cos α, (a) (a) Side view of reconstructed 3D points
630 36 (b) (b) Vertical view of reconstructed 3D points 9 Fig. 9 Reconstructed 3D points cloud (b) (b) Vertical view of reconstructed vertical lines 12 Fig. 12 Results of reconstructed vertical lines (a) (a) Side view of reconstructed lines 10 Fig. 10 The detected images of vertical lines in three-dimensional space (a) (a) Side view of reconstructed horizontal lines (b) (b) Vertical view of reconstructed lines 13 Fig. 13 Results of reconstructed lines (b) (b) Vertical view of reconstructed horizontal lines 11 Fig. 11 Results of reconstructed horizontal lines (a) (a) Side view of reconstructed points and lines (a) (a) Side view of reconstructed vertical lines (b) (b) Vertical view of reconstructed points and lines 14 Fig. 14 Fusion of reconstructed points and lines
5 : 631 2. 15,, 24, 12, 10, 1 600 1 200.,,. 16. 1 3.,,. 17,,,..,,,. 18 19,,. (a) (12 3 ) (a) Three of twelve images captured before axial motion Fig. 17 17 The first image for measurement (b) (12 3 ) (b)) Three of twelve images captured after axial motion 15 Fig. 15 Indoor scene images (a) (a) Side view of reconstructed points and lines Fig. 18 18 The second image for measurement (b) (b)) Vertical view of reconstructed points and lines 16 Fig. 16 Fusion of reconstructed points and lines Fig. 19 19 The third image for measurement
632 36 2 3,,,. Table 1 1 Ratio of distances between adjacent corners on calibration board 378 1 0.0091 0.0457 0.0120 Table 2 2 Results of the measurement ( ) ( ) ( ) ( ) (%) 1 2 215 216.88 1.88 0.88 2 3 255 258.90 3.90 1.53 4 5 210 213.43 3.43 1.63 5 6 300 302.45 2.45 0.82 7 8 310 296.94 13.06 4.21 8 9 230 220.85 9.15 3.98 10 11 600 594.67 5.33 0.89 Table 3 3 Results of the measurement ( ) ( ) ( ) ( ) (%) 1 2 650 621.34 28.66 4.41 2 3 250 240.32 9.68 3.87 4 5 1 030 1 029.60 0.40 0.04 6 7 650 628.73 21.27 3.27 8 9 1 290 1 298.70 8.70 0.67 10 11 300 286.84 13.16 4.39 10 12 795 785.64 9.36 1.18 11 13 800 809.33 9.33 1.17, 5 %,,.,. 6,,,,.,,..,,. References 1 Sequeira V, Goncalves J G M, Ribeiro M I. 3D reconstruction of indoor environments. In: Proceedings of International Conference on Image Processing. Lausanne, Switzerland: IEEE, 1996. 405 408 2 Wang R, Luebke D. Efficient reconstruction of indoor scenes with color. In: Proceedings of the 4th International Conference on 3D Digital Imaging and Modeling. Banff, Canada: IEEE, 2003. 402 409 3 Debevec P E, Taylor C J, Malik J. Modeling and rendering architecture from photographs: a hybrid geometry-andimaged-based approach. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. New Orleans, USA: ACM, 1996. 11 20 4 Gibson S, Howard T. Interactive reconstruction of virtual environments from photographs, with application to sceneof-crime analysis. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology. Seoul, Korea: ACM, 2000. 41 48 5 Hu B, Brown C. Interactive indoor scene reconstruction from image mosaics using cuboid structure. In: Proceedings of the Workshop on Motion and Video Computing. Orlando, USA: IEEE, 2002. 208 213 6 Tomasi C, Kanade T. Shape and motion from image streams under orthography: a factorization approach. International Journal of Computer Vision, 1992, 9(2): 137 154 7 Polleyfeys M, Gool L V, Vergauwen M, Verbiest F, Cornelis K, Tops J. Visual modeling with a hand-held camera. International Journal of Computer Vision, 2004, 59(3): 207 232 8 Polleyfeys M, Nister D, Frahm J M, Akbarzadeh A, Mordohai P, Clipp B. Detailed real-time urban 3D reconstruction from video. International Journal of Computer Vision, 2008, 78(2-3): 143 167 9 Shi J, Tomasi C. Good features to track. In: Proceedings of Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 1994. 593 600 10 Wang Z H, Wu F C, Hu Z Y. MSLD: a robust descriptor for line matching. Pattern Recognition, 2009, 42(5): 941 953 11 Triggs B, Mclauchlan P F, Hartley R, Fitzgibbon A W. Bundle adjustment a modern synthesis. In: Proceedings of the International Workshop on Vision Algorithms: Theory and Practice. London, UK: Springer, 1999. 298 372 12 Lourakis M, Argyros A. The Design and Implementation of a Generic Sparse Bundle Adjustment Software Package Based on the Levenberg-Marquardt Algorithm, Technical Report 340, Institute of Computer Science-FORTH, Greece, 2004 13 Zhang Z Y. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11): 1330 1334
5 : 633 14 Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91 110 15 Lhuillier M, Quan L. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(3): 418 433 16 Hartley R. In defense of the eight-point algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(6): 580 593 17 Nister D. An efficient solution to the five-point relative pose problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(6): 756 770 18 Hartley R, Zisserman A. Multiple View Geometry in Computer Vision. UK: Cambridge University Press, 2002 19 Fitzgibbon A W, Cross G, Zisserman A. Automatic 3D model construction for turn-table sequences. In: Proceedings of the European Workshop on 3D Structure from Multiple Images of Large-Scale Environments. Freiburg, Germany: Springer, 1998. 155 170. 2002. 2006... E-mail: whyzf2005@163.com (ZHANG Feng Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. He received his bachelor degree from Qufu Normal University in 2002 and his master degree from Dalian University of Technology in 2006. His research interest covers 3D reconstruction and virtual reality. Corresponding author of this paper.).. E-mail: lmshi@nlpr.ia.ac.cn (SHI Li-Min Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. His research interest covers image processing and 3D reconstruction.).. E-mail: fmsun@163.com (SUN Feng-Mei Associate professor at the Faculty of Sciences, North China University of Technology. Her research interest covers optics and intelligent signal processing.).. E-mail: huzy@nlpr.ia.ac.cn (HU Zhan-Yi Professor at the Institute of Automation, Chinese Academy of Sciences. His research interest covers camera calibration, 3D reconstruction, Hough transform, and vision guided robot navigation.)