A Multi-Camera Network System for Markerless 3D Human Body Voxel Reconstruction

Transcription

1 A Multi-Camera Network System for Markerless 3D Human Body Voxel Reconstruction Tao Yang Yanning Zhang Meng Li Dapei Shao Xingong Zhang Shaanxi Key Laboratory of Speech & Image Information Processing School of Computer Science, Northwestern Polytechnical University Abstract This paper presents a fully automated system for realtime 3D human visual hull reconstruction and skeleton voxels extraction. The main contributions include: (1) A novel network based system is presented, which uses AXIS network cameras as video capture device, and performs a parallel processing among data capture, 3D voxel reconstruction and display. (2) A new human visual hull reconstruction algorithm is given. This approach firstly segments the foreground accurately by an efficient Gaussian Mixture Model (GMM) and a shadow model in HSV color space, then extends the standard Shape-From-Silhouette (SFS) algorithm with online Region-of-Interest (ROI) estimation and binary searching, and finally construct skeleton probability visual hull with distance transform. Experiments with real video sequences show that the system can process eleven 6x4 video sequences at a frame rate of 15fps, and construct human body voxels reliably in complex scenarios with cast shadows, various body configurations and multiple persons. 1. Introduction Developing a real-time and fully automatic human motion capture system is a subject of great scientific and commercial interest [1-8], and it finds application in a variety of fields such as bio-mechanical analysis [1], humancomputer interfaces [2], film and game industries, and intelligent video surveillance. Typical motion capture mechanisms require that people be instrumented with several markers. The motion of people is then reconstructed from the positions of markers. Not only such kind of system is prohibitively expensive, but the people do not feel comfortable with cumbersome markers. In recent years there has been interest in multi-view marker-less motion capture system. Those system and these systems basically use Shape-From-Silhouette or visual-hull technique to construct the 3D shape of the object. The idea of using silhouettes for 3D shape reconstruction was first introduced by Baumgart in his PhD thesis in Since then, different variations of the SFS paradigm have been proposed, and now SFS has become a popular 3D reconstruction method for both static objects and dynamic objects. Cheung et al. [8] present an efficient sparse occupancy reconstruction technique and use a cluster of PCs to fit ellipsoids in real-time on the reconstructed voxels, and refine the visual-hull by temporal visual-hull alignment [1]. Caillette et al.[7] present a hierarchical statistical framework to improve the reconstruction speed, and keep color information in every inline voxel. Although SFS has many advantages, such as the silhouettes are easy to obtain and process, the implementation of algorithm is also uncomplicated, and it is suitable for the reconstruction of both static and dynamic objects, its performance still suffer from the following drawbacks: (1) When the number of silhouettes is few, the shape estimated will be very rough, and a coarse model is one major disadvantage to the following work such as the acquisition of human kinematic models and marker-less motion tracking. (2) The accuracy of the construction result is also depended on the foreground silhouette segmentation result, which may easily be influenced by environmental noises like shadows of moving object. (3) The standard SFS approach uses a unique sampling rate for visual hull reconstruction in the entire 3D zone, when object move around in a large area, it needs to trade off between the sampling rate and processing speed. This paper presents a fully automated multi-view markerless motion capture system. The network based system framework is designed to be open and scalable. In the system s motion capture component, we firstly implement efficient GMM model segment object s silhouette and use color information to remove shadows. Then a novel online binary search based SFS algorithm is presented to reduce the searching space in visual hull construction, and to move the Region-of-Interest (ROI) follow a person. Finally, we

2 AXIS IP Cameras ĂĂ Motion JPEG from IP Cameras Video Recorder Server(VRS) Power over Ethernet(PoE) Switch Multi-Camera Self Calibration Video Recorder Server MYSQL Database Server 3D Scene Viewer Motion Capture PCs MYSQL Database Server Motion Capture Motion Capture PCs 3D Scene Viewer (a) System hardware architecture (b) System software architecture Figure 1. Architecture of our markerless motion capture network system. propose an distance transform based algorithm to generate human skeleton probability visual hull, which is suitable for further estimation and tracking of human kinematic model. The rest of this paper is organized as follows: Section 2 introduces the setup of the network based motion capture system. Section 3 describes the algorithm of motion capture component. Section 4 presents experimental results in scenes with cast shadows, various body configurations and multiple persons. The paper is concluded in Section System overview The ultimate goal of our system is to operate in realtime with convenient wire or wireless IP camera network, support precisely multiple people motion capture simultaneously. These requirements bring great challenges such as robustness to time synchronization, environment noises and real-time processing Architecture Our infrastructure is composed of nine AXIS 211M and two AXIS 211W wireless digital network cameras in a laboratory environment. We use a video recorder server (VRS) to store the network videos, a MYSQL database server to preserve the video information, foreground silhouette image paths and camera calibration parameters, several analysis system components for foreground segmentation and visual hull reconstruction, and a user interface to show the motion capture results in a 3D virtual scene. Fig.1 shows the software and hardware architecture of the system. The overall system architecture is designed to be open and scalable, and it s very convenient to add more cameras, video analysis PCs. Thanks to the network system framework, the video analysis component can run on one or several PCs, retrieve live video data from the video server, and store the motion capture results in the local host database or remote server. The 3D user interface component access and display the live Figure 2. Screen shot of the video recorder server network video, foreground silhouette and human skeleton voxels from the VRS and the MYSQL server. All the VRS, MYSQL server, multiple video analysis modules and user interfaces can run simultaneously System components Video Recording The VSR server is configured with Intel Core 2 Duo E 3.0GHz CPU, 2GB RAM, and 0GB hard disk, and runs under windows XP operation system. We record video from eleven AXIS IP cameras [5] that provide video as Motion JPEG via HTTP. The Motion JPEG is chosen since it is more suitable for seeking frames at different times, which is important for both online and off-line analysis. Another reason is that if multiple users access one AXIS camera at the same time, the video frame rate will decrease, while our design can support video access from various user interface PCs simultaneously. Currently, we use one VRS to record video from eleven

3 IP cameras at 30 frames per second (fps) at 6x4 resolutions. This design results in about 51 GB per camera per 24 hours, requiring close to 0 GB per day in storage for 12 cameras. Fig.2 shows the screen shot of our video recorder server with eleven live videos. Video Synchronization and Access Because the IP camera doesn t support hardware synchronization, we design the following strategy to solve this problem.during preserving the Motion JPEG pictures, the computer time of the VSR accurate to the millisecond is assigned to every saved image as its unique identification, and saved into the MYSQL database. Because we preserve multi-view videos independently at 30fps, the maximal time difference between two video streams is 33.3 milliseconds. In practice, we found that the maximal time difference is usually less than 10 milliseconds. When the video analysis component tries to access the live image of a certain camera, the MYSQL database server will return image with the closest time. Experiment results demonstrate that the proposed synchronization strategy is acceptable for further human visual hull reconstruction. Camera Calibration and 3D viewer The cameras are calibrated using Tomas s Matlab toolbox [9]. In our experiment, a set of virtual 3D points is made by waving a laser pointer through the working volume, and its projections are found with sub-pixel precision and verified by a robust RANSAC analysis.calibration experiment results show that with our software based video synchronization strategy, Tomas s approach [9] is able to calibrate eleven IP cameras and reach about 0.4 pixel reprojection error. The precision is enough for further human Visual-Hull reconstruction and skeleton tracking. In order to display the live camera video and human visual hull in a 3D virtual scene, we have developed a 3D viewer by Microsoft Direct3D. 3. Human Body Voxel Reconstruction 3.1. Foreground Silhouette Segmentation The first step in processing is segmenting people from the background. We use an extension of the Gaussian mixture model (GMM) for pixel level background modeling. Gaussian Mixture Background Model The most popular GMM for the foreground segmentation is proposed by Stauffer and Grimson [10], the pixel intensity density is estimated with mixture of Gaussians: P (X t ) = K ω i,t η(x t, µ i,t, i,t) (1) i=1 η(x t, µ i,t, i,t) = 1 (2π) n 2 i,t 1/2 e 1 2 (Xt µi,t)t i,t 1 (X t µ i,t) (2) where µ i,t is the estimate of mean and i,t = σi,t 2 I is the estimate of the variance that describe the ith Gaussian components at time t. For computational reasons the covariance matrices are kept isotropic. Given a new pixel sample X t at time t the recursive update formula are: ω i,t = (1 α)ω i,t 1 + αm k,t (3) µ i,t = (1 ρ i )µ i,t 1 + ρ i X i,t (4) σ 2 i,t = (1 ρ i )σ 2 i,t 1+ρ i (X i,t µ i,t ) T (X i,t µ i,t ) (5) where ρ i = αη(x t µ i, σ i ). For each input frame, the Gaussians are ordered by the value of ω/σ,which increases both as a distribution gains more evidence and as the variance decreases. Then the first B distributions are chosen as the background model: B = arg min b ( b ω k > T ) (6) k=1 where T is the minimum portion of the background model. The original mixture of Gaussian background model can deal with slow lighting changes, periodic motions from a cluttered background and long term scene changes. But its processing speed is influenced by the number of Gaussian components, and it cannot handle shadows. To improve the processing speed of GMM model, the method in [11] is implemented in our system. This algorithm similar to the standard Stauffer and Grimson algorithm [10] with additional selection of the number of the Gaussian components for each pixel, and achieves a very fast foreground segmentation with slightly wore performance. Cast Shadow Removal Since shadow points are often misclassified as object in the background subtraction step, and will cause server errors in foreground silhouette segmentation and visual hull construction, thus it s critical to detect and remove shadow for accurate object detection in video streams. In order to deal with shadows, many algorithms have been presented in the literature, and a comprehensive survey of moving shadow detection approaches can be found in [12], which also includes comparative evaluation of the existing approaches. To develop a robust shadow removal algorithm that best suitable for our system, we implemented two different kinds of shadow detection algorithm, include the method in [13] which uses normalized cross-correlation of local region intensity, and the method in [14] which works with color similarity analysis in HSV color space. Because the system

4 is installed in an indoor laboratory environment with relative simple background color and lack of texture information (see Fig.2), after comprise the above two approaches by extensive experiments, we finally select color based approach [14] for its efficiency and low false alarm rate while maintain the high detection rate. The key component of the shadow detection algorithm is briefly introduced as follows: After background subtraction, a shadow mask SPk for each (x, y) point is calculated by the following formula: ifα XV k (x,y) Bk V (x,y) β 1 (Xk S SP k (x, y) = (x, y) BS k (x, y)) τ S Xk H(x, y) BH k (x, y) τ H F k (x, y) = 1 0 otherwise (7) where Xk V (x, y) and BV k (x, y) are the pixel values at coordinate (x,y) in the input image and in the background model at frame k, F k (x, y) denotes the moving foreground mask. The main reason to work in HSV color space is that this space corresponds closely to the human perception of color and it has proven more accurate in distinguish shadows than the RGB space. The choice of parameters in formula (7) is based on the following rules. The use of α and β allow us to find false alarm pixels where the background illumination value is slightly decreased by shadows. The setting of parameter τ S and τ H is done empirically with the assumption that the color and saturation of shadowed and non-shadowed pixels does not vary too much. After cast shadow detection, we imply connected component analysis to fill holes inside the human body and remove small foreground blobs as noises. Various experiment results of the foreground segmentation and shadow removal are given in Section Efficient Visual Hull Construction The standard algorithm for computing a visual-hull is to start from voxels in 3-D space, project them into the available image planes and keep only those which lie inside all the silhouettes of the object. ROI Extraction and Binary Searching To calculate a precise human visual hull, a high sampling rate is needed in the reconstruction algorithm. Because people often move in a large area in our system, instead of using a unified sampling rate in the entire 3-D space, we present a novel binary search based visual hull construction algorithm, which can automatically select the ROI for each individual, and search inside the ROI efficiently. The following are the processing steps of the proposed algorithm: Firstly, the entire space of motion capture is divided into discrete NxNxN voxels and represented by vector H. Then we scan the space H with a large searching interval, and classify the searching space into two categories: background and ROI by formula (8): { H(i) = ROI if Adjacent(i) > 0 Background otherwise (8) where Adjacent(i) represents the state of eight adjacent points of voxel I, and has value of one as foreground and zero as background. Finally, a binary search algorithm (as shown in Table 1) is presented to check the voxels only inside the ROI. 3D Skeleton Probability Visual Hull Construction Since our goal is to present an algorithm for 3D human visual hull reconstruction that is best adapted for multipeople skeleton estimation and markerless motion capture, it s more desirable to kept more usefully information in the human voxel. Suppose we have a precise 3D human skeleton, and a common fact is that its projection in each camera view is also the skeleton in the object s silhouette. Thus in this paper, we extend the traditional SFS algorithm with distance transform based skeleton estimation. Distances transform, or called as distance map, are a 2D array of the same size as the camera view. The map supplies each pixel of the image with the distance to the nearest boundary pixel in a binary silhouette F: DT (p) = min k {d(p, q k ) : F (q k ) = 0} (9) where d(p, q k ) denotes a distance metric of pixel p and qk, and k lists all pixels in the image F. In our system, we

5 0 0 (b) (a) Figure 4 Online Region-of-Interest extraction results.the background voxels are labeled with green color, and the ROI is displayed with red color. Note that the proposed algorithm segment the ROI with multiple objects correctly, and greatly reduce the searching space. use calculate the distance map DTj for each camera view j, and then construct a new 3D skeleton probability visual hull with accumulating the foreground distance transform in multiple camera view. The skeleton probability visual hull reconstruction results are given in Section Experimental Results The proposed algorithm and network system are being used in our laboratory. In this section, some examples demonstrate the effectiveness of the algorithm for foreground segmentation and shadow detection, efficient searching and 3D skeleton probability visual hull construction are given. Currently, we use one PC to segment two 6x4 video sequences at a frame rate of 30fps, and transform the processing results to the database server. Fig.3 shows the foreground segmentation results. The first row contains the original input images of four views. The second row shows the foreground mask of efficient GMM model [11], note that the human arm and leg are seriously affected by the cast shadow (Fig.3, second row). The third row displays the shadow detection result using formula (7). We chose the following parameters empirically as α = 0.98,β = 0.8, τs = 0.1,τH = 0.05 which are proved to be robust in our system. By integrating color and illumination information, most of the shadows are corrected detected (Fig.3, third row, red points) and removed(fig.3, fourth row). View 1 Figure 3 View 2 View 3 View 4 Foreground segmentation results. Fig.4 shows examples of the ROI extraction results with single (Fig.4a) and multiple people (Fig.4b), in which the green and red color denotes the background and ROI respectively. Note that the proposed algorithm can segment the ROI with various numbers of objects correctly, and greatly reduce the searching space. The 3D space of motion capture in our system is divided into 1x1x1 discrete voxels, and the entire searching space contains 8,,1 voxels. In Table 2 we compare the number of searched voxels with our approach and the standard SFS algorithm. Thanks to the proposed online ROI extraction and binary searching algorithm, the computational cost of our approach is far lower than the standard SFS, and

6 Science Foundation (No ), and Cultivation Fund of the Key Scientific and Technical Innovation Project, Ministry of Education of China(No.7085). References (a) 1 (b) Figure 5 Visual hull construction results. the average relative searched points ratio is only 0.72% of the traditional approach using a uniform sampling rate (Table 2, bottom row). Fig.5 shows the skeleton probability visual hull construction results. Note that the proposed algorithm correctly labels the skeleton voxels with bright colors with various body configurations and multiple persons (Fig.5b). 5. Conclusions This paper presents a system performing robust and realtime 3D human visual hull reconstruction. The novelties include an open and scalable network camera based markerless motion capture system, and an efficient skeleton visual hull construction algorithm using ROI extraction, binary searching and distance transform. The presented results confirm the validity of the proposed algorithm and system. The future work will focus on 3D visual hull based real-time human body tracking and pose estimation. 6. Acknowledgement This work is supported by the National High Technology Research and Development Program(863) of China(No.09AA01Z315), National Natural Science Foundation of China (No ), China Postdoctoral [1] K. Cheung, S. Baker, and T. Kanade, Shape-from silhouette across time part I: Theory and algorithms, International Journal of Computer Vision,62(3): ,05. [2] B. Rosenhahn, C. Schmaltz, T. Brox, J. Weickert, D. Cremers,and H.P. Seidel, Markerless motion capture of manmachine interaction, IEEE Conference of Computer Vision and Pattern Recognition, pp.1-8,08. [3] B.Michoud, E.Guillou, and S.Bouakaz, Real-Time and Markerless 3D Human Motion Capture Using Multiple Views, Human Motion-Understanding, Modeling, Capture and Animation,pp ,07. [4] M. Niskanen, E. Boyer, R. Horaud, and I. Oulu, Articulated motion capture from 3-D points and normals, IEEE Transactions on Pattern Analysis and Machine Intelligence,pp ,09. [5] C.K. Wan, B.Z. Yuan, and Z.J. Miao, Markerless human body motion capture using Markov random field and dynamic graph cuts, Visual Compute,pp.373-3,08. [6] B.C. Shen, H.C. Shih, and C.L. Huang, Real-time Human Motion Capturing System, IEEE International Conference of Image Processing,pp ,05. [7] F. Caillette and T. Howard, Real-Time Markerless Human Body Tracking with Multi-View 3-D Voxel Reconstruction, British Machine Vision Conference, volume 2,pp.597-6, 04. [8] K. M. Cheung, T. Kanade, J.Y. Bouguet, and M. Holler, A real time system for robust 3D voxel reconstruction of human motions, In CVPR,pp.714-7,00. [9] T.Svoboda, D.Martinec, and T.Pajdla, A convenient multi-camera self-calibration for virtual environments, PRESENCE: Teleoperators and Virtual Environments, 14(4),pp.7-422,05. [10] C.Stauffer, E.Grimson, Learning Patterns of Activity Using Real-Time Tracking, IEEE Transactions on Pattern Recognition and Machine Intelligence, 22(8): ,00. [11] Z. Zivkovic, F. van der Heijden, Efficient adaptive density estimation per image pixel for the task of background subtraction, Pattern Recognition Letters,27, pp.773-7, 06. [12] A. Prati, I. Mikic, M. Trivedi, R. Cucchiara, Detecting Moving Shadows: Formulation, Algorithms and Evaluation, IEEE Transactions on Pattern Recognition and Machine Intelligence, vol.25, no.7, 03. [13] Y.L.Tian, M.Lu, A. Hampapur, Robust and efficient foreground analysis for real-time video surveillance, In Computer Vision and Pattern Recognition Conference,pp , 05. [14] R. Cucchiara, C. Grana, M. Piccardi, and A. Prati, Improving shadow suppression in moving object detection with HSV color information, IEEE Conference on Intelligent Transportation Systems,pp ,01.