A Research on Moving Human Body Detection Based on the Depth Images of Kinect

Transcription

1 Sensors & Transducers 2014 by IFSA Publishing, S. L. A Research on Moving Human Body Detection Based on the Depth Images o Kinect * Xi an Zhu, Jiaqi Huo Institute o Inormation and Communication Engineering, Beijing Inormation and Technology University, No. 35, Beisihuan Middle Road, Chaoyang District, Beijing, , China * Tel.: * Zhuxian1962@163.com Received: 20 December 2013 /Accepted: 28 February 2014 /Published: 31 March 2014 Abstract: Moving human body detection is a signiicant research direction in the ields o image processing and computer vision. A new method o moving human body detection is proposed in this paper based on Kinect, which extracts depth images with Kinect, so that the moving object in which people are interested can be obtained without useless background inormation. Five consecutive rames o depth images are used to operate the calculation o three-rame dierencing twice, which is called dual-three-rame dierencing algorithm in this paper. Then combined with improved Sobel operator based on our directions, the ull edge o moving human body can be obtained. Experiments demonstrate that the proposed algorithm is simply to operate, easy to implement, and can get the whole moving human body without morphological processing. Copyright 2014 IFSA Publishing, S. L. Keywords: Moving human body detection, Kinect, Depth image, Three-rame-dierencing, Sobel operator. 1. Introduction In recent years, moving object detection, tracking, identiication and analysis based on video stream is crucial in computer vision domain, including bank monitoring, intelligent transportation monitoring and pedestrian tracking, etc. The vast majority purposes o algorithm aims at operating on human action. With people's social security consciousness gradually increasing, video monitoring has got more and more attention. In typical automatic monitoring applications, people oten use a ixed camera to supervise the scene. Moving object detection means to separate interested area rom the scene in original image sequences, and detecting moving object accurately in the above applications as well as systems o target tracking, man-machine interaction and precision guiding is a very important task. So ar, there are three traditional methods to detect moving object: optical low, background subtraction and rame dierence. In 1950, Gibson proposed the concept o optical low, which reers to inormation o movement o pixels in the images relected on the retinal plane [1]. Optical low is a method o dynamic analysis on the velocity vector characteristics o each rame o the image pixels. According to the changes o optical low vectors caused by moving object, the moving object can be detected. But it has a complex calculation, a slow speed o image processing and it s easy to be aected by many kinds o noise. Background subtraction is most commonly used in moving target detection, which makes dierential operation between the current rame and background model so that the dierent part can be deined as moving object [2]. However, i background changes, it will inluence 30 Article number P_1947

2 detected results. Frame dierencing method is to make dierential calculation with contiguous two or three rames to get the moving target. It has simple algorithm, easy implementation, and is suitable in conditions o camera static or environment changing. It s diicult to ind out the pixel points that do not change signiicantly, so sometimes there are phenomenon o holes and double shadows [3]. On the one hand, as Kinect is not aected by light, environment changing, texture, shadow, color scale and other actors, and the pixel values o depth images are only associated with the relative position o target, what s more, depth images extracted rom Kinect are robust with less noise than color images. Kinect can satisy the requirement o real-time depth inormation low processing with the speed o 30 rames per second. On the other hand, the edge o image is not susceptible to the eects o noise and brightness mutation [4]. According to the problems caused by three-rame dierencing, such as isolated noise, holes, shadows and so on, this paper proposes an improved moving human body detection method which combines dual-three-rame dierencing algorithm and Sobel operator based on our directions, to overcome the above problems and has got remarkable results. 2. Algorithm o Moving Human Body Detection 2.1. Introduction and Theory o Kinect In June 2009, E3 announced the device o Kinect or the irst time, the code name o which is Project Natal. On June 6th 2010, Microsot oicially released motion-sensing peripheral device o XBOX360, namely Kinect, together with a series o games, such as Star Wars Kinect Sports, dancing games published by MTV and so on. Kinect, shown in Fig. 1, mainly makes use o sensing chip PS1080 and equipment released by PrimeSense company. an inrared CMOS camera as the right part, which extracts depth image with resolution o , depth sensing range o 0.4 to 7 meters [5], best sensing range o 1.2 to 3.5 meters; in the middle o Kinect, there is an RGB camera used to get color streams; there is a multi-array microphone at the back o Kinect which has a unction o speech recognition, iltering environmental noise and eliminating echo; the base part is called motorized tilt, which can adjust proper angles according to the target, with the physical tilt angle range o -27 to 27, vertical angle o view 43, and horizontal angle o view 57. The application o Kinect has been involved in multiple areas. For example, virtual itting room, namely, customers only need to stand in ront o Kinect with some proper gestures to choose their desiring clothes, so that the dressed person will be displayed in the screen, without the trouble that people have to it or many times; in the teaching o medical experiments, auxiliary teaching work can be done with Kinect to identiy some common actions; in video meeting, instead o mouse and keyboard, human's gestures can be detected through Kinect to realize the interace operation over a long distance; the movement o wheelchair can be controlled with the recognition o gesture based on Kinect, et al. The hardware core o Kinect is PS1080 System-on-Chip rom PrimeSense Company, which has a powerul parallel computing logic to control near inrared source, use images to code and cast near inrared ray. Kinect is based on the theory o Light Coding [6], namely it uses inrared emitter to launch near inrared rays in the whole detecting area; and then codes or the area under test which means to produce laser speckles; transmits laser speckles with the chip; inally, makes use o inrared ray CMOS camera to receive and decode relected rays to extract depth images. The theory o laser speckle shows that when laser cast on rough objects or rosted glasses, random and discrete diraction spots are ormed. They are dierent in size and shape according to the distances rom the depth sensor, so that we realize marking objects. Ater marking objects within the scene, a reerence plane is picked or every ixed distance to record the laser speckles. When an object is detected, the three dimension inormation is acquired ater a series o matching, multiplying, interpolating operation. The low diagram o acquiring depth images rom Kinect is shown in Fig Principle Description Fig. 1. Appearance and composition o Kinect. Depth sensor is comprised o an inrared emitter displayed as the let part in Fig. 1, and The algorithm presented in this paper is composed o our parts, namely, depth images extraction, median iltering, dual-three-rame dierencing and improved Sobel operator. First o all, install Kinect or Windows sdk, extract moving human depth images and store them in PC. In the second step, operate median ilter on every 31

3 rame o depth images and then use dual-three-rame dierencing method or every contiguous ive rames. Thirdly, get edge inormation with improved Sobel operator and combine the result with the last step, so that we can obtain precise moving human outline. The low block diagram is shown in Fig. 3. Emit near inrared rays, produce laser speckles, mark Select reerence planes, record speckles Select speckle patterns on the surace o objects as test patterns Match test pattern with reerence pattern, calculate correlation coeicient Calculate and get 3-D inormation Output depth images Fig. 2. Flow diagram o acquiring depth images rom Kinect. Depth images rom Kinect Median ilter i 1,iand i,i+ 1dodo dierence operation operation separately separately to obtain to dobtain andd 1 2 Binarization processing i 2,iand i,i+ 2dodo dierence operation separately to to obtain d andd 3 4 Use Sobel operator to extract depth image edge B A B ' ' ' ' (d 1&d 2) (d 3& d 4) to to obtain A Obtain moving human outline Fig. 3. Diagram o algorithm Extracting Depth Images with Kinect Ater installing Kinect or Windows sdk and corresponding driver, depth images are extracted and stored when people act in Visual Studio The thinking in C# is shown as ollows: Step 1: create a new project, add Microsot. Kinect, System. IO, System. Media and corresponding reerence, append Image Control and other Controls about words to MainWindows Form. Step 2: initialize the device o Kinect and extract depth stream. Step 3: map depth images to the corresponding color images. Step 4: deine a threshold; i the dierences between current rame and previous rame surpass the threshold that means some actions happen, store the current rame to avoid some redundant inormation and save the system memory at the same time. Step 5: set the saved path and save the satisactory images to this ile Dual-Three-Frame Dierencing Method Frame dierencing means that calculating the dierence between contiguous two rames can get the moving object. However, the obtained moving 32

4 object is usually bigger that its actual size with shadows and this method can t detect overlapping region or the reason o holes [7]. Improved on two-rame dierencing, three-rame dierencing needs continuous three rames as a group, obtained d 1 and d 2 through two-rame dierencing, then makes an operation o d 1 & d 2. But when object moves very slowly, there may be three still consecutive rames that cause to lose the target. To overcome this situation, this paper proposes a method called dual-three-rame dierencing algorithm with consecutive ive rames, to make dierencing processing. Speciic algorithm process is shown in Fig. 4. Depth image i 1,i and i,i 1do do dierence operation separatelyto to obtain d andd, and, + do dodierence dierence operation separately to obtain + i 2 i i i d andd 3 4 d&d 1 2 d 3&d4 d d 5 6 Fig. 4. Dual-three-rame dierencing algorithm Improved Sobel Edge Operator Edge, with which we can realize image segmentation, reers to mutations in gray level, structure or other inormation, which is not only an end o a region, but also a beginning o an area. Sobel operator is a irst derivative edge detection operator, including two 3 3 matrices, one o which is used to detect horizontal edges and the other is used to get vertical outlines [8]. Every pixel in images does convolution sum operation with 3 3 templates, then edges can be extracted according to a proper threshold. The convolution templates o original Sobel operator are depicted in Fig. 5. The expression o the gradient amplitude ollows: M = S + S, (1) 2 2 x y where the templates o S x and S y are shown as ollows: Compared with Prewitt operator, Sobel operator calculates the weights or the inluence o pixel locations to reduce the og-level o edge, so the edge is more distinct. Isotropic Sobel, another orm o Sobel operator, has two templates, too [9]. One is used to obtain horizontal edges; the other is used to get vertical edges. But compared with original Sobel operator, the position weighted coeicient o Isotropic Sobel operator is more accurate, what s more, the magnitude o gradient keeps the same when detecting dierent directions o edges. Usually, Isotropic Sobel operator is got through exchanging all o 2 in original Soble operator matrices with 2. In Sobel operator, the eects o neighborhood pixels to the current pixel are not equivalent, so dierent distances have dierent weights and the contribution results o operator are dierent too. Compared with Roberts operator and Prewitt operator, Sobel operator has a better extraction result. The advantages o Sobel operator is simple and ast calculation, but it only has two directional templates which can detect horizontal and vertical outlines, lost much useul inormation. To improve detection eects, two 3 3 templates are added to Isotropic Sobel operator, respectively 45 and 135 gradient direction templates, shown in Fig. 6 [10]. The edge Si (, j ) is obtained as ollows: Fig. 5. Original Sobel operator templates. Si (, j) A B C D = (2) 33

5 According to the templates in Fig. 6, we can compute the ollowing ormula: A= X( i+ 1, j 1) + 2 X( i+ 1, j) + X( i+ 1, j+ 1) Xi ( 1, j 1) 2 Xi ( 1, j) Xi ( 1, j+ 1) (3) Kinect or Windows sdk, in the environment o Windows 7 PC, depth images are gained. The results based on the algorithm proposed by this paper are depicted in Fig. 9, where (h) and (i) are depth images, and (j), (k) are their edge results. B = X( i 1, j+ 1) + 2 X( i, j+ 1) + X( i+ 1, j+ 1) Xi ( 1, j 1) 2 Xij (, 1) Xi ( + 1, j 1) (4) C = 2 X( i, j+ 1) + X( i+ 1, j+ 1) X( i+ 1, j) 2 X( i 1, j) (5) D= 2 X( i 1, j) + X( i 1, j+ 1) + 2 X( i, j+ 1) 2 X( i, j 1) X( i+ 1, j 1) 2 X( i+ 1, j) (6) (a) (b) (a) Gradient direction templates o 0 (let) and 90 (right) (c) (d) Fig. 7. Comparisons o edge operators. (b) Gradient direction templates o 45 (let) and 135 (right) Fig. 6. Templates o our directions. 3. Experimental Results and Discussion Firstly, experiments are carried out in the circumstance o Matlab on a running video rom Weizmann database, separately using Canny operator, Sobel operator, improved Sobel operator based on our directions to extract edges, the results o which are shown in Fig. 7, where (a) is original color image, (b) is original Sobel operator, (c) is Canny operator, and (d) is improved Sobel operator. Then operate three-rame dierencing algorithm (e), dual-three-rame dierencing algorithm () and the improved dual-three-rame dierencing method (g) separately on the same running video to realize moving human body detection, the results are illustrated in Fig. 8. Thirdly, ater installation o (e) () (g) Fig. 8. Comparisons o dierencing methods. Experimental results show that extracting human body with Kinect has a good advantage o other cameras, because it can ilter uninterested environment easily, avoid ollowing steps such as morphological processing and extract human body only reerring to the distances rom depth sensors. With the improved dual-three-rame dierencing, we can get human edge inormation accurately. 34

6 Acknowledgements This paper was supported by Inormation system integration and deep developing technology or CMB ground concentrating transportation, Grant No. 2011ZX Reerences (h) (j) Fig. 9. Edge extraction based on depth images with improved dual-three-rame dierencing. 4. Conclusions (i) This paper presents a moving human body detection algorithm based on depth images, which combines improved dual-three-rame dierencing and Sobel operator, and applies to depth images extracted rom Kinect. Firstly, depth images escape rom the intererence o some uninterested background inormation, as well as background modeling and other cumbersome steps; secondly, operate dual-three-rame dierencing on contiguous ive rames o depth images and use improved Sobel operator to extract the edge o moving human, then combine the above two results in order to gain the complete moving human outline. The algorithm proposed has a good robustness. It not only has a small amount o calculation and a ast speed, but also extracts precise moving human edges and eliminates the phenomenon o holes and shadows caused by rame dierencing method, acilitating urther processing o movement characteristics. (k) [1]. Gao Tao, Liu Zheng-Guang, Yue Shi-Hong, et al., Robust background subtraction in traic video sequence, Journal o Central South University o Technology, 17, 1, 2010, pp [2]. Lian Xiaoeng, Dai Jun, Liu Zaiwen, Wang Xiaoyi, et al., Research on the moving body detection based on video stream, Journal o Bei Industry and Commerce University, 27, 6, 2009, pp [3]. Zhang Wen, Li Rong, Zhu Jianwu, Target detection algorithm based on Gaussian mixture model and three-rame dierencing, Modern Electronics Technique, 35, 8, 2012, pp [4]. Tong Nian-Nian, Duan Xiao-Hui, A research on a detection algorithm or vehicle monitoring system, The 1 st Chinese Conerence on Intelligent Surveillance, Beijing, 2002, pp [5]. John Stowers, Altitute. Control o a Quadrotor Helicopter Using Depth Map rom Microsot Kinect Sensor, in Proceedings o the IEEE Interaction Conerence on Mechatronics, 2011, pp [6]. Garcia J., Valencia E. S., Zalevsky Z., Ha ayin I. L., Range mapping using speckle decorrelation, US Patent No B2, October 7, [7]. Haiying Zhang, Kun Wu, A Vehicle Detection Algorithm Based on Three-rame Dierencing and Backround, in Proceedings o the 5 th International Symposium on Computational Intelligence and Design (ISCID), 2012, pp [8]. Yuan Chunlan, Xiong Zonglong, et al., Research o image edge detection based on Sobel operator, Laser Inrared, 39, 1, 2009, pp [9]. Wu Yan, Yang Lei, An improved orientation ilterbased algorithm or ingerprint enhancement, Journal o Huazhong University o Science and Technology (Nature Science Edition), 35, 2, 2007, pp [10]. Lin Chunli, Classiication o human actions and anomaly detection based on video sequences, Harbin Engineering University, Copyright, International Frequency Sensor Association (IFSA) Publishing, S. L. All rights reserved. ( 35