Real-time Top-view People Counting Based on A Kinect and NVIDIA Jetson TK1 Integrated Platform

Transcription

1 2016 IEEE 16th International Conference on Data Mining Workshops Real-time Top-view People Counting Based on A Kinect and NVIDIA Jetson TK1 Integrated Platform Guangqin Li, Peng Ren, Xinrong Lyu, and He Zhang College of Information and Control Engineering China University of Petroleum (East China) Qingdao, China ligq@s.upc.edu.cn, pengren@upc.edu.cn, lvxr@upc.edu.cn, hezhang@s.upc.edu.cn Abstract In this paper, we describe how to establish an embedded framework for real-time top-view people counting. The development of our system consists of two parts, i.e. establishing an embedded signal processing platform and designing a people counting algorithm for the embedded system. For the hardware platform construction, we use Kinect as the camera and exploit NVIDIA Jetson TK1 board as the embedded processing platform. We describe how to build a channel to make Kinect for windows version 2.0 communicate with Jetson TK1. Based on the embedded system, we adapt a water filling based scheme for top-view people counting, which integrates head detection based on water drop, people tracking and counting. Gaussian Mixture Model is used to construct and update the background model. The moving people in each video frame are extracted using background subtraction method. Additionally, the water filling algorithm is used to segment head area as Region Of Interest(ROI). Tracking and counting people are performed by calculating the distance of ROI center point before and after the frame. The whole framework is flexible and practical for real-time application. Keywords-Kinect; Jetson TK1; Top-view; People counting; Waterfilling. I. INTRODUCTION People counting is an important task for marketing research [1] (e.g. pedestrian traffic management [2], tourists flow estimation and people flow monitoring [3] and security applications [4], [5] (in evacuation, estimating people flow is essential to save lives). In the literature, most high precision algorithms of people counting are performed on computers [6]. However, in some scenarios such as on a bus, the space for placing equipments are limited. Therefore, how to develop real-time people counting frameworks on a small-sized embedded device is highly demanded. In the past few years, novel algorithms were rarely implemented via embedded platforms. It was because that the previously popular embedded development platforms [7], [8] have limited resources. Most of these embedded platforms could not meet the real-time requirements. Therefore, the implementation of sophisticated counting algorithms on these embedded platforms are technically difficult. Nowadays, with the recent development of embedded technologies, there are many embedded computing platforms which can be performed in parallel operation. Raspberry Pi, BeagleBone Black and Jetson TK1 are currently popular embedded development platforms on the market, which are widely used and more powerful. Especially Jetson TK1 is armed with rich interfaces and powerful computing capabilities. In this paper, we set up a top-view camera for recording walking people because occlusion is naturally avoided. Most of popular stereo cameras [9], [10] are mainly using Kinect for windows version 1.0, Kinect for windows version 2.0 (hereinafter referred to as Kinect V2), Primesense and etc. Kinect V2 is a latest camera, which can collect higher resolution image and range accuracy. Since the heads are always closer to the camera than other parts of the body in top view of Kinect sensor [11], detecting heads is posed as a problem of finding the suitable local minimum regions in the depth image. The depth image [1] is shown in Fig. 1. When some people walk in the counting area, they are very close to each other and partial occlusion might occur. In this situation, depth images play a more important role in people detection and extraction than color images because color features are rarely capable of identifying occlusions. In the literature, counting by detection is the most popular approach. Once people have been detected, the counting problem is solved by numbering the detected objects. However, object detection also has two unsolved problems. Firstly, one object may correspond to different bounding boxes and currently the commonly used NMS (non-maximum suppression) [12] for deciding a bound box tends to become invalid in complex situation where many people have interactions with each other. Secondly, some objects are too difficult to be detected, because they may be partially or even wholly occluded. Both problems affect the counting performance. Our framework adapts water filling method which is motivated by the water filling process [12]. It is a good solution to the above two problems arising in detection. In the water filling process, the water moves away from the heave and out to the nearby hollow under the force of gravity until the gravitational potential energy cannot be reduced any more. We simulate the rain by generating the raindrop according to an uniform distribution. Once /16 $ IEEE DOI /ICDMW

2 Range of Depth 8.0m Range of Detection 4.5m 255 Kinect 2.0 Jetson TK1 Camera 0.5m 0 Pixel Value Set Up Communication Platform Figure 1. Kinect view range (left) and example depth image (right) a raindrop arrives, we compare its landing spot with its neighborhood and find the descent direction until it cannot descend any more, then the number of raindrop at the balance spot increases. Water filling algorithm can detect the top of the head more accurate. II. PROPOSED SYSTEM In this section we introduce our proposed system for people-counting. First, a communication channel between camera and computing platform is built to obtain the depth image. Second, the depth image processing algorithm is developed to achieve people counting. The overview of proposed system is shown in Fig. 2. A. Hardware platform structures Our video capture device is a Microsoft Kinect for Windows version 2.0. Kinect V2 is an upgraded version of Kinect for windows version 1.0, which is integrated with depth sensors, color sensors and infrared sensors. Compared to the prevision version, Kinect V2 has a lot of improvements. For example, the color image resolution is promoted from to , the depth of the image resolution is promoted from to , and the detection range is more accurate. Our computing platform is NVIDIA s production Jetson TK1. Jetson TK1 runs on NVIDIA s latest Tegra TK1 SoC (system on chip). It has four 32-bit ARM Cortex- A15 general purpose cores running at 2.3GHz, a low-power Cortex-A15 shadow core that runs when the system is not under heavy load, and 192 Kepler GPU cores running at 852MHz for graphical horsepower. In addition to the powerful image processing capabilities of Tegra TK1, we employ the undersampling strategy and propose a statistical tracking algorithm for real-time image processing. Jetson TK1 is capable of performing GPU computation via NVIDIA s CUDA (Compute Unified Device Architecture) computing platform. Jetson TK1 is a powerful platform for computer vision and CUDA applications and may have potential applications in robotics, medicine, security, automotive, and even national defense. On the other hand, the configuration for operating kinect V2 is as follows: Windows 8.1 or higher Figure 2. Acquire Images Background Model Water Filling Head Detection People Tracking And Counting The proposed people counting system Microsoft Visual Studio 2013 Community Edition with Update 2 or higher Microsoft.NET Framework 4.5 [13] Kinect for Windows SDK 2.0 [13] USB 3.0 Though both Jetson TK1 and Kinect V2 have their own merit for real-time data collection and processing, they are not compatible to each other. To establish an embedded platform integrating Kinect V2 and Jetson TK1, we build a communication channel between them. The details for build the communication channel is described as follows: Step 1): Initialization video capture device on PC. In order to use the Kinect V2 to collect data for Jetson TK1, we first initialize Kinect V2 by a Windows 8.1 PC. This should be done before physically connecting the Kinect V2 to the Jetson TK1 for the first time. We download the Kinect drivers and the Kinect for Windows SDK 2.0 [13], 469

3 Figure 3. Download toolkits Install tools Compile Sample Micro USB Micro USB Internet Jetson TK1 Flash OS Install tools Test compiled sample Jetson TK1 configurations via communications with a PC from the Kinect official website and install them according to the installation guide. We then connect kinect V2 with PC through USB 3.0 and running an example in the installation directory. If some images can be captured by the Kinect sensor and illustrated on the PC screen, the initialization is successfully. Step 2): Systematic configuration of the Jetson TK1. Jetson TK1 is an embedded platform with the operating system Linux Ubuntu on board. However, the raw Linux system have limited ability of maneuvering the hardware resources on board for data processing. To enhance the data processing power of the platform, we need to perform a systematic configuration of the board, and install some toolkits that are used for manipulating GPUs and developing signal processing algorithms, etc. NVIDIA corporation provide a setup package for configuring Jetson TK1, which is Jetson TK1 Development Pack. Jetson TK1 Development Pack includes host (Ubuntu Desktop) and target (Jetson TK1) development tools, APIs and packages (OS images, tools, CUDA, OpenCV, APIs, middleware, samples, documentation) for developing with NVIDIA Tegra K1 on the NVIDIA Jetson TK1 platform. The whole configuration process requires an iterative interplay between a PC with Linux Ubuntu System and the NVIDIA Jetson TK1 through an USB 3.0 connection. Additionally, both of them require an internet access. The overview of this process is shown in Fig. 3. Once we finish this step, we disconnect the Jetson TK1 from the PC. The Jetson TK1 will then operate as an individual platform and will not be connected to PC again in our task. Step 3): Acquire depth and color images. We connect the Kinect V2 to the Jetson TK1 via USB 3.0. Before we exploit it as our embedded platform for processing the data captured by the Kinect V2, we need to install some other toolkits, such as Libfreenect2, Libusb and TurboJPEG, to Jetson TK1. Libfreenect2 is a crossing platform driver of Kinect V2. Libfreenect2 provide a lot of APIs that can be used to control the camera. Libusb is a C library that gives applications easy access to USB devices in many different operating systems. TurboJPEG APIs wraps libjpeg-turbo and provides an easyto-use interface for compressing and decompressing JPEG images in memory. These toolkits are downloaded directly to the Jetson TK1 via the on board Linux Ubuntu The installation of these toolkits enable the Jetson TK1 to obtain color and depth video smoothly from the Kinect camera. The integration of Kinect V2 and Jetson TK1 thus obtained forms an infrastructural platform which can potentially implements image/video processing algorithms in real-time on board. The Kinect V2 is possible to capture high resolution RGB and depth images/videos, and the GPUs in Jetson TK1 are capable of conducting parallel computing for algorithms, which in turn possibly result in real-time outputs. B. People counting scheme In this subsection, we introduce the people counting method, which is mainly divided into three parts. 1) Build the background model using Gaussian Mixture Model [10]. 2) Detect people head using water filling method [12]. 3) Apply people tracking and counting. 1) Background model and foreground extraction: Based on the infrastructural platform established in the previous subsection, we start developing people counting algorithms. We commence by developing a motion detection strategy for segmenting the moving objects from depth images using background subtraction technique. The change of the illumination or motion has strong influence on the intensity of color image but not on the depth information. Therefore, we use Gaussian Mixture Model (GMM) [10] for background modeling which can be achieved by computing its probability as follows: P (z t )= L ω i,t η(z t,μ i,t, ) (1) i,t i=1 where z t represent one pixel in the original image containing Z pixels. ω i,t and L is the weighted Gaussian and the number of Gaussian, respectively. η is the probability of Gaussian function defined by: η (z t,μ, ) = 1 (2π) Z e 1 2 (zt μ t ) T 1 (zt μ t ) Once background is constructed, whereafter foreground will be obtained. In the foreground region, there may be a foreground object overlapping, then head detection is used to separate the overlapping targets. (2) 470

4 2) Head detection: According to [12], we exploit the water filling algorithms for head detection. Compared to alternative detection algorithms (e.g. Mean Shift and Random Forest), water filling has more advantages in antiinterference and detection of children. we represent the depth value of pixel (x, y) byf(x, y). Due to the noise of depth image, f(x, y) can be discontinuous or even null-valued. This method presents a measure function g(x, y) to measure f(x, y). g(x, y) is defined if and only if ε>0, which is an error controlling parameter (x i,y i ), (x j,y j ) subjected to (x i x j ) 2 +(y i y j ) <ε,iff(x i,y i ) f(x j,y j ): f(x i,y i )+g(x i,y i ) f(x j,y j )+g(x j,y j ) g(x i,y i ) g(x j,y j ) g(x i,y i ) 0, g(x j,y j ) 0 The function g(x, y) has three advantages: Robustness: Handle noises in the original depth data. Locality: Detect people in different heights. Scale-Invariance: Detect people head at any scales. The form of function f(x, y) can be seen as a land with humps and hollows. The raindrop in the hump will flow directly to the neighborhood hollow under force of gravity. Step by step, the hollow region will gather a lot of raindrops. The function g(x, y) reflects the quantity of raindrop at (x, y). After the rain stops, the regions with a lot of rain drop can be classified as a hollow. Water filling algorithm can be seen as the simulation of the above process. Furthermore, the water filling approach solve the basic problem that the quantity of raindrops is a continuous value and different raindrops may reach the land simultaneously, which lead to the difficulty of simulating their interactions. Therefore, water filling algorithm is an effective strategy for head detection. Supposed that there is a foreground object contour image patch with the size of M N, its depth function is f(x, y) and initialize the measure function as g(x, y) =0.LetT be the threshold, K be the number of raindrops. Provided (x m,y n ) is one of the neighbor of (x, y). Then we search the potential human head pixel (ˆx, ˆy) by minimizing the following score function O: min O(x m,y n )=f(x m,y n )+g(x m,y n ) f(x, y)+g(x, y) (3) which enables to obtain the minimum value at the point (ˆx, ˆy). IfO(x m,y n ) is greater than zero, g(x, y) increases one and the loop is end, otherwise take point (x m,y n ) as a new point (x, y) and perform the above operations again. After all the K raindrops are set to their stable places, we obtain the final measure function g(x, y). Then thresholding and contour analysis operations are applied to g(x, y). By setting threshold T on g(x, y), contour analysis is employed to find contours corresponding to each people. Finally, take A B B A Figure 4. A 1, A 2, B 1 and B 2 demonstrate the center of each rectangular for two person in two successive frames, d 11, d 12, d 21 and d 22 represent the distance between A 1, A 2, B 1 and B 2, respectively. all the contours extracted as the input of people tracking and counting. 3) People tracking and counting: In this subsection, a simple and effective method was proposed to track the head position and count people. In addition, people s walking direction (i.e. in or out) can be judged by the tracking scenario. Fig. 4 illustrate the overview of the proposed people tracking and counting process. We first find the center of each depth image patch for different person. For instance, A i and A j demonstrate the two different persons in the previous frame. While B i and B j represent the two head patches in current depth image. We then calculate the pairwise distance of head patches above as follows: d ij = (x i x j ) 2 +(y i y j ) 2 (4) where the x { } and y { } exhibit the coordinates of the center associated with corresponding depth patch. By calculating the distance, it is obviously that the same depth patch of one person possess the less distance among the previous frame and the current frame than that with another. Fig. 6 illustrate the threshold selection in our calculating approach. Data B represents the distance of one person among the current and previous frame, while Data A is the distance between different people on the same frame. Thus, it is obviously obtain the threshold which leads to a high performance of distinction for different people. By setting a suitable distance threshold, when the d ij distance is smaller than the threshold, A i is the same path of B j in current depth image. Only if the tracking scheme continued for more than twelve frames, the counter increases one for one person. If the distance is larger than the distance threshold value, we refer to the rectangle of the depth patch as a new people. III. EXPERIMENTAL RESULTS We conducted three groups of comparative experiments in order to compare with other algorithms and platforms. The 471

5 Figure 5. Proposed system detection results Figure 7. DSP system detection results Figure 6. Threshold selection first one is the comparison of different hardware platforms. The second one is the comparison of different algorithms. The third one is to test the accuracy of proposed system for real-time stream of people counting. A. Experimental comparisons on alternative infrastructural platforms The work presented in [8] was conducted on a DSP embedded platform for people stream statistics. They proposed a people counting method for estimating the number of people crowd in large spaces such as lobbies, wholesaleand super-stores. The their main algorithm flow was as follows: 1) Capturing frame from camera; 2) Constructing background model; 3) Updating background model; 4) Extracting foreground; 5) Constructing the weighting model; 6) Accumulating the weighed pixels and counting people. In order to facilitate comparison, we remove the tracking part in our experiments. We perform this set of experiments on our Jetson TK1 which is equipped with 32-bit ARM Cortex-A15 (2.3GHz), 192 Kepler GPU (852MHz) and 16GB memory. The results are shown in Fig. 5 and Fig. 7. In the experimental results, it is shown that our results are in better agreement with the real statistics. Especially, for the first fifty frames, the DSP embedded platform has a bigger error in people statistics. B. Experimental comparisons based on alternative algorithms This set of experiments are performed on a PC machine with Intel Xeon E5 CPU (3.5GHz) and 64GB memory for comparing algorithm performances. The water filling algorithm is compared with the a commonly used people counting scheme presented in [10] based on depth and color images, whose procedure is as follows: 1) Capturing depth and color image from Kinect; 2) Motion detection blob analysis; 3) People segmentation use U-disparity and color matching; 4) People tracking and counting. The experimental results are shown in Table 1. According to Table 1, it is obvious that the accuracy of the water filling algorithm is significantly higher than the algorithm proposed in [10]. Two groups of experiments were run in the same environment, which ruled out other factors except the algorithm. C. Experiments on proposed system This set of experiments are performed on the Jetson TK1 in a practically captured depth sequences. The first 300 frames of the image are used for training the background model. After this training procedure, the people counting starts operating. The people counting experimental results are shown in Fig. 8. The resulted statistics of proposed system on two different databases are demonstrated in Table 2. IV. CONCLUSION In this paper, we have constructed an embedded people counting system which physically integrates Kinect V2 and 472

6 Figure 8. People detection and counting results Jetson TK1 and algorithmically exploits a water filling scheme. We have presented a communication strategy for transferring data from Jetson TK1 to Kinect V2. We have adapted a water filling algorithm for human head detection and proposed an effective tracking and counting scheme. Experimental comparisons performed on various platforms with alternative algorithms have validated the superiority of our proposed system. Through different experimental setups, it has been shown that the proposed system has great potential applications in various practical situations. REFERENCES [1] T.-H. Chen, T.-Y. Chen, and Z.-X. Chen, An intelligent people-flow counting method for passing through a gate, in 2006 IEEE Conference on Robotics, Automation and Mechatronics. IEEE, 2006, pp [2] C. Hong, J. Yu, D. Tao, and M. Wang, Image-based threedimensional human pose recovery by multiview localitysensitive sparse retrieval, IEEE Transactions on Industrial Electronics, vol. 62, no. 6, pp , [3] C. Hong, J. Yu, J. Wan, D. Tao, and M. Wang, Multimodal deep autoencoder for human pose recovery, IEEE Transactions on Image Processing, vol. 24, no. 12, pp , [4] N. Bernini, L. Bombini, M. Buzzoni, P. Cerri, and P. Grisleri, An embedded system for counting passengers in public transportation vehicles, in Mechatronic and Embedded Systems and Applications (MESA), 2014 IEEE/ASME 10th International Conference on. IEEE, 2014, pp [5] A. B. Chan, Z.-S. J. Liang, and N. Vasconcelos, Privacy preserving crowd monitoring: Counting people without people models or tracking, in Computer Vision and Pattern Recognition, CVPR IEEE Conference on. IEEE, 2008, pp [6] X. Zhao, E. Delleandrea, and L. Chen, A people counting system based on face detection and tracking in a video, in Advanced Video and Signal Based Surveillance, AVSS 09. Sixth IEEE International Conference on. IEEE, 2009, pp [7] A. G. Vicente, I. B. Munoz, P. J. Molina, and J. L. L. Galilea, Embedded vision modules for tracking and counting people, IEEE Transactions on Instrumentation and Measurement, vol. 58, no. 9, pp , [8] H.-C. Chen, Y.-C. Chang, N.-J. Li, C.-F. Weng, and W.-J. Wang, Real-time people counting method with surveillance cameras implemented on embedded system, in Proceedings of the World Congress on Engineering and Computer Science, vol. 1, [9] B. Alessandro, D. S. Luigi, and A. Pietro, People tracking using a time of flight depth sensor, in 2006 IEEE International Conference on Video and Signal Based Surveillance. IEEE, 2006, pp [10] C. Wateosot, N. Suvonvorn et al., top-view based people counting using mixture of depth and color information., in The second asian conference on information systems, ACIS. Citeseer, [11] T. Van Oosterhout, S. Bakkes, and B. J. Kröse, Head detection in stereo data for people counting and segmentation. in VISAPP, 2011, pp [12] X. Zhang, J. Yan, S. Feng, Z. Lei, D. Yi, and S. Z. Li, Water filling: Unsupervised people counting via vertical kinect sensor, in Advanced Video and Signal-Based Surveillance (AVSS), 2012 IEEE Ninth International Conference on. IEEE, 2012, pp [13] A. Jana, Kinect for windows SDK programming guide. Packt Publishing Ltd,