A Lidar-based 4D scene reconstruction system

Transcription

1 A Lidar-based 4D scene reconstruction system Csaba Benedek and Zsolt Jankó MTA SZTAKI, Distributed Events Analysis Research Laboratory & Geometric Modeling and Computer Vision Laboratory SPAR Europe and European LiDAR Mapping Forum (ELMF), Amsterdam, The Netherlands

2 integrated 4D (i4d) System i4d: a pilot system for reconstruction and visualisation of complex spatio-temporal scenes by integrating two different types of data: o outdoor 4D data measured by a Velodyne LIDAR sensor, o and 4D models of moving actors obtained in a 4D studio. Velodyne LIDAR 4D Studio

3 4D scene reconstruction Goal: creating viewpoint free videos Lidar (Deva) Recorded camera image 4D reconstruction studio (GMCV) Synthetized 4D scene sketch

4 Velodyne LIDAR

5 Velodyne LIDAR Hardware: Velodyne HDL-64E LIDAR Output: 2.5D point cloud sequence from outdoor environments Technical data: 64 laser and sensor 120 m distance <2cm accuracy >1.333M points/sec

6 Surveillance: Courtyard scenario by fixed LIDAR LIDAR on a mobile mapping platform

7 Surveillance scenario Lidar data flow Foregroundbackground separation FG BG Pedestrian detection & multi target tracking Environment reconstruction Integration & visualization of spatiotemporal model Actor videos pre-recorded in a 4D studio Building 4D models of walking pedestrians

8 Foreground detection Velodyne cylinder projection 3D point cloud Frontal range image part: Velodyne sensor Full view range image (64x1024 pixels): 8

9 Foreground - background modeling Background: Temporal Mixture of Gaussians (MoG) model: o Noisy result - errors in textured or dynamic background f bg p k s w i s η μ σ t t ε bg s f bg p s i Foreground class: non-parametric kernel density model Background Foreground k s t ε fg p ζ r N s ε τ k d s t d r t h

10 Dynamic MRF model 2-D pixel lattice graph: S = {s} Nodes: image points (s is a pixel) Edges: interactions cliques o intra-frame edges: spatial smoothness o inter-frame edges: temporal smoothness MRF energy function Temporal smoothness term E V D ω α ω ω β ω ω Data term s S s S r N s s S r N s Energy optimization o Graph cut based method (real time) Spatial smoothness term

11 Label backprojection Point cloud labeling based on the segmented range image o Problems due to angle quantization for the discrete pixel lattice o Misclassified points near object edges and, shadow edges Smart backprojection o Expliting contextual information in label backprojection

12 Foreground - evaluation

13 Pedestrian separation and tracking Detection: ground projection + blob separation Detection Assignment Kalman filt. prediction Kalman filt. correction Tracking: state machine

14 Reconstruction of the background 2.5D point cloud 2D panorama photo 3D surface model 3D textured scene model

15 Virtual pedestrians - 4D studio

16 i4d Project MTA SZTAKI Output of the integrated model

17 i4d workflow

18 Registered Lidar and camera sensor

19 Multi target tracking and person re-identification based on LIDAR

20 Multi target tracking and person reidentification based on LIDAR

21 Moving LIDAR platform Horizontal LIDAR: street object and traffic monitoring DiFiLTON-ARC Tilted LIDAR: reconstruction of building facades

22

23 Lidar Input point cloud frames (1,2,,N) Grid based segmentation of each point cloud (1, N) Point cloud registration Surface reconstruction Moving object detection and classification Large planar regions Other objects Grid based resegmentation and connected component analysis (merged cloud) Merged cloud Tree crown removal

24 Preprocessing point cloud segmentation A grid based method. o Uniform grid defined in the 2D space along the ground plane. o The grid is segmented as an image first o Runs in real time. Point classes: o Noise and sparse data: grid cells with a few data points o Ground surface: cells of points with small elevation differences (used threshold: 25cm) o Tall objects (e.g. walls): cells with large elevation differences (more than 310cm) or large maximal elevation (used 350cm) o Short street objects: everything else (cars, pedestrians, street furniture, etc)

25 Street scene segmentation Color codes: road wall vehicle+ street objs. O. Józsa, A. Börcs and Cs. Benedek: Towards 4D Virtual City Reconstruction From Lidar Point Cloud Sequences, ISPRS Workshop on 3D Virtual City Modeling, Regina, Saskatchewan, Canada, May 28-31, 2013, vol. II-3/W1 of ISPRS Annals of Photogrammetry, Remote Sensing and the Spatial Information Sciences pp , 2013 A. Börcs, O. Józsa and Cs. Benedek: Object Extraction in Urban Environments from Large-Scale Dynamic Point Cloud Datasets, IEEE International Workshop on Content-Based Multimedia Indexing (CBMI), Veszprém, Hungary, June 17-19, 2013

26 Registration Only the points in the Wall or Tall Static Object class are used. o Noise and dynamic data are removed o Reduced number of points o Remaining points are strong features Registration techniques o Normal Distributions Transform(NDT) used for most of the following results o Trimmed Iterative Closest Point algorithm (TrICP, Chetverikov at all, ICV 2005) alternative method used in some tests

27 Registration: results Frame #1 Frame #2 30 merged frames Frame #3

28 On the streets of Budapest Our office BME Central building Kende utca (MTA SZTAKI)

29 Kálvin square

30 Tree crown removal Overhanging trees can corrupt object detection Registered data is dense, thus sparse regions with large scattering (such as leafs) can be detected Overhanging tree crowns can be removed

31 Result of upper vegetation detection

32 Vehicle detection Frame #1 30 merged frames Frame #2 Frame #3 2D recognition 3D backprojection

33 Distinguishing moving vs. static objects Moving objects result in blurred blobs in the merged cloud Solution: preserving the time stamp information for each point

34 Separating moving and parking vehicles Center point of the point cloud in the individual time frames Center point sequence for a moving vehicle Center point sequence for parking vehicles

35 Analysing motion tracks Pedestrians: Turning vehicle: Trajectory of point cloud centers Point cloud sequence color = time stamp 11/6/2013 MTA SZTAKI / EEE 35

36 Road mark detection Road marking detection (zebra crossing) Vertical histogram Ground points after intensity based threshold Horizontal histogram

37 Surface reconstruction Poisson triangulation of the obtained point cloud Figures: main and southeastern facades of the Great Market Hall

38 NDT vs TrICP NDT TrICP NDT is more robust for featureless buildings (like office houses) NDT TrICP TrICP gives superior results for surfaces containing characteristic features.

39 NDT vs TrICP NDT is more robust for featureless buildings (like office houses) TrICP gives superior results for surfaces containing characteristic features. NDT TrICP

40 Surface models Budapest, Kende utca

41 Surface + texture Great Market Hall, Budapest

42 Data fusion Roof (aerial) + facades (terrestrial scan) Aerial scans Infoterra Hungary Ltd

43 Mixed reality

44 4D scenario in front of the Great Market Hall

45 Working with Topcon datasets LadyBug camera Sick Lidar Output: colored point cloud + panoramic images

46 Distance measurement on panoramic images

47 Aerial Lidar scans Astrium (Infoterra) HU

48 Vegetation filtering by echo number information Optical aerial image LIDAR echo map Astrium (Infoterra) HU Astrium (Infoterra) HU Astrium (Infoterra) HU

49 Scene segmentation and traffic analysis from LIDAR data Pointcloud Detected vehicles Segmented pointcloud Data source: Astrium (Infoterra) HU Detection of vehicles and vehicle groups

50

51 Publications [C8] Cs. Benedek, Z. Jankó, Cs. Horváth, D. Molnár, D. Chetverikov and T. Szirányi: An Integrated 4D Vision and Visualisation System, International Conference on Computer Vision Systems, St. Petersburg, Russia, Lecture Notes in Computers Science, Springer, 2013 [C7] A. Börcs, O. Józsa and Cs. Benedek: Object Extraction in Urban Environments from Large-Scale Dynamic Point Cloud Dataset, IEEE International Workshop on Content-Based Multimedia Indexing (CBMI), Veszprém, Hungary, June 17-19, 2013 [C6] O. Józsa, A. Börcs and Cs. Benedek: Towards 4D Virtual City Reconstruction From Lidar Point Cloud Sequences, ISPRS Workshop on 3D Virtual City Modeling, Regina, Saskatchewan, Canada, May 28-31, 2013, to appear in ISPRS Annals of Photogrammetry, Remote Sensing and the Spatial Information Sciences [C5] Cs. Benedek, D. Molnár and T. Szirányi: A Dynamic MRF Model for Foreground Detection on Range Data Sequences of Rotating Multi-Beam Lidar, International Workshop on Depth Image Analysis, Tsukuba City, Japan, November 2012, Lecture Notes in Computers Science, Springer, 2013 [C4] J. Hapák, Z. Jankó, D. Chetverikov. GPU-Based Real-Time Spatio-Temporal Reconstruction Studio. In Proc. 28th Spring Conference on Computer Graphics, ACM, Smolenice, Slovakia, pp , [C3] J. Hapák, Z. Jankó, D. Chetverikov, "Real-Time 4D Reconstruction of Human Motion", Proc. 7th International Conference on Articulated Motion and Deformable Objects (AMDO 2012), Mallorca, Spain, Lecture Notes in Computer Science, Springer, vol. 7378, pp , [C2] C. Blajovici, D. Chetverikov, Z. Jankó, 4D Studio for Future Internet: Improving Foreground- Background Segmentation, IEEE International Conference on Cognitive Infocommunications, Kosice, Slovakia, 2012 [C1] D. Chetverikov, L. Hajder, Z. Jankó, C. Kazó, J. Hapák Multiview 3D-4D Reconstruction at MTA SZTAKI, IEEE International Conference on Cognitive Infocommunications, Kosice, Slovakia, 2012

52 Acknowledgement Funding: The i4d Project ( ) is funded by the internal R&D grant of MTA SZTAKI, Budapest, Hungary Participating laboratories of MTA SZTAKI: o Distributed Events Analysis Research Laboratory o Geometric Modeling and Computer Vision Laboratory Contributing people: DEVA Lab.: Csaba Benedek, Attila Börcs, Csaba Horváth, Oszkár Józsa, Gábor Mészáros, Dömötör Molnár, Balázs Nagy, Tamás Szirányi GMVC Lab.: Dmitry Chetverikov, Iván Eichhardt, Zsolt Jankó