Robust Human Tracking using Statistical Human Shape Model with Postural Variation

Size: px

Start display at page:

Download "Robust Human Tracking using Statistical Human Shape Model with Postural Variation"

Juliet Clark
5 years ago
Views:

1 Robust Human Tracking using Statistical Human Shape Model with Postural Variation Kiyoshi Hashimoto, Hirokatsu Kataoka and Yoshimitsu Aoki Keio University Yuji Sato Panasonic Corporation Abstract Human tracking in monocular image sequences has been studied in the field of computer vision for many kinds of applications such as surveillance system, intelligent room, sports video analysis and so on. Human tracking in real environment is challenging topic due to various factors such as illumination change, partial or almost complete occlusion of human body, and wide variety of body shapes. In this paper, we present a robust human tracking using statistical human shape model of appearance variation with postural change. Our part-based statistical human model can generate learned appearances of main human poses, and enables effective and robust human tracking with simple features such silhouette, edge and color. Our proposed method achieves human tracking robust not only to partial occlusion but also to postural change. The experimental results validate the robustness of our methods in the real indoor environments. I. INTRODUCTION The technique of real-time and robust human action recognition in real environment such as public institutions and commerce facilities is being counted on applications in a wide variety of fields including surveillance systems, human interface and so on [1]. While tracking a person, the appearance and posture are always changing. These variations of human often make human tracking fail. In order to solve this problem, many tracking methods have been proposed in recent years to improve its robustness. The previous methods [2], [3] applied color information to human appearance model. Color information is insusceptible to illumination change and viewpoint. In the case of existing similar objects, these methods fail to track easily. Particle filtering is known as effective object tracking approach in complicated situations such as occlusion. Particle filtering tracks objects by iteration of next state prediction and likelihood evaluation and widely used for human tracking frameworks [6], [8]. Nummiaro [4] uses HSV color space which is robust to illumination change. Okuma et al. [5] and Wu et al. [7] separate human region into each body part and create body part classifiers from a large number of training images. These methods are robust to a wide variety of body shapes and clothing. To separate human region into each body have advantages such as being easy to handling partial occlusion, resistant to view point changes and pose variations. Those methods, which perform rectangle-based human region tracking, focus on only walking or standing people as tracking targets. Tracking failure tends to increase for non-standing states, such as sitting, bending or other postural human. And we can only extract the position and scale of tracking humans from an estimated rectangle. In this sense, postural change must be considered in human tracking methods to evade this problem. Tracking methods which can extract rough pose information in addition to position are desirable for more detailed activity recognition. In our proposed method, we focus on the appearance variation with postural change. We created two low-dimensional human shape models with principal component analysis. These models can represent shape variations of each body part efficiently. And our system can capture various postures flexibly. It is difficult for previous rectangle-based methods to track humans who are sitting or bending. We present a novel human tracking method using this low-dimensional human model. The posture model allows us to capture a human variation in realscene. This tracker also enables us to extract the position, shape of body parts and posture from a flexible human region. As a result, even simple human action recognition becomes possible. II. PROPOSED METHOD We propose two statistical models of human-like body shape to track the human even when the postural change occurs. The first model is the Active Ω Model (AOM) which represents the relation between contour of head-shoulder (Ωshape) and shoulder joint points. This model can track head poses more detailed. The head-shoulder part of humans typically has a distinctive omega-like shape in almost all view angles. This is used for human detection and tracking. Wu et al. propose a head-shoulder detection method based on edgelet features and tracking is achieved by data association of detection results [13]. In [14], a rapid and robust Ω-shape detection and tracking technique based on boosting of local HOG features is proposed and shows good performance in real scenes. However, these methods use local features and AdaBoost classifier. Calculating high-dimensional features is time-consuming. Learning classifiers and preparing training images is burdensome task. AOM is created by only few hundreds Ω-shape data. The second is a Main-parts Link Model (MLM) to track each body part robustly. We need to estimate human pose to recognize human activities. Felzenszwalb propose an efficient framework for part-based modeling and recognition of objects. In [15], representing an object by a collection of parts arranged in a deformable configuration allow for qualitative descriptions of visual appearance. MLM is more simple than pictorial /13/$ IEEE 2478

Fig. 1. A part of Ω-shape dataset. Each Ω-shape data is represented by 20 head-shoulder contour points and 2 shoulder joint points. All data is normalized with center of head as a reference point.

Next, we perform head detection using AOM. If reliable head is detected, we initialize each body part and start tracking the human integrate model of both AOM and MLM.

L OW- DIMENSIONAL H UMAN S HAPE M ODELS Active Ω Model (AOM): This is a low-dimensional model to represent a correlation between the head-shoulder contour (omega-like shape) and the position of

2 Fig. 1. A part of Ω-shape dataset. Each Ω-shape data is represented by 20 head-shoulder contour points and 2 shoulder joint points. All data is normalized with center of head as a reference point. structures model and suitable for the problem treated in this paper. We detect and track humans using these low dimensional human shape models. Firstly, background subtraction is performed. Next, we perform head detection using AOM. If reliable head is detected, we initialize each body part and start tracking the human integrate model of both AOM and MLM. Detail of each process is mentioned in later chapters. III. L OW- DIMENSIONAL H UMAN S HAPE M ODELS Active Ω Model (AOM): This is a low-dimensional model to represent a correlation between the head-shoulder contour (omega-like shape) and the position of shoulder joints. AOM is learned according to the following procedure. AOM represents the head-shoulder contour with 22 points (Fig.1 right). These Ω-shape data are normalized with center of head as a reference point. We extract 154 Ω-shape data manually from headshoulder images in all kinds of view points. Principal Component Analysis (PCA) compresses 44 dimensional data (a point includes x and y coordinates) from Ω-shape to effective dimensions. We defined 4 dimensions from above as AOM. This PCA subspace represent more than 90% of cumulative contribution ratio. This model space that has 4 parameters includes the variations of Ω-shape (Fig.4 center). The most likely head-shoulder shape can be estimated by results of fitting AOM the head-shoulder in a real image within a lowdimensional subspace. This information is very important for pose estimation and activity recognition. For example, if the distance of between two shoulder points is narrow, the person may turn sideways. We can get many useful information from Ω-shape estimated in image. Main-parts Link Model (MLM): This model represents a constraint of each body part connection (head, torso and leg) in low-dimensional PCA subspace. As a result of this model, we can put into practice human-like shape tracking. Fig.2 is an example of postural data. In our system, human posture is modeled with 5 parts (head 1, upper body 2, lower body 2). Each body part has 5 parameters, which are position, width, height and angle (x,y,w,h,θ). These postural data has 25 dimensions. And we analyzed them with PCA as well as AOM. The PCA result is shown in Fig.4 left. PCA subspace which has 3 dimensions is derived from 94 postural data. The postural connection between each body part is included in this subspace. Since MLM can represent the postural change due to 2479 Fig. 2. A part of postural dataset. Each postural data is represented by 5 rectangles which has 5 parameters (position, width, height and angle). human0 s rotate, movement or direction, it can capture human region flexibly under postural change scenes. Our tracking system outputs body part connections that indicate postural status. Hence, we can track the pose roughly by positions and orientations with five parts, which enables us to analyze the temporal motion of each part. Integrated Model of AOM and MLM: Finally we integrated this two models. For aligning the relative position between two models, we move the base point of AOM to the center of head rect in MLM. For aligning the angles of two models, we rotate AOM in keeping with head rect angle of MLM. This definitive human shape model is low dimensional (7 dimensions) and effectively represent the variation of mainparts and Ω-shape. We track humans by fitting this integrated model to consecutive images in a video. IV. H UMAN D ETECTION Most of the human detection methods use feature extraction and machine learning. In these methods, we have to prepare a number of human images. Then, we extract edge or texture information by feature descriptor and we create classifiers to learn these features. We can do these task in advance since they need not on-line information from image sequence. But it is not easy to prepare high-quality training image dataset. And the more we represent shape of object in detail, the more high dimensional feature descriptor is required to learn the appearance variation sufficiently. Visual feature extraction, which tends to be a time-consuming process, is also need for on-line tracking. Hence, detection with these highly computational classifiers is time-consuming. In our system, human detection is performed to detect head-shoulder contour using Active Ω Model. It works with low computational cost since we need to perform fitting lowdimensional AOM model with simple features. The flow of human detection is shown in Fig.3. Input image is split into blocks in Fig.3(a). In each block, we randomly create a lot of AOM samples of various position, shape and scale in Fig.3(b). We evaluate each AOM by brightness orientation and silhouette of background subtraction. This likelihood evaluation process is explained in detail in the next chapter.

(a) divide into blocks (b) search for various Ω-shape (c) detect Ω-shape (d) initialization to track Fig. 3. The flow of head detection. (a) Input image is split into 20 20 blocks.

Only when moving object is in the block to use background subtraction information, we keep on searching for optimal AOM until the reliable Ω-shape is found.

3 (a) divide into blocks (b) search for various Ω-shape (c) detect Ω-shape (d) initialization to track Fig. 3. The flow of head detection. (a) Input image is split into blocks. (b) In each block, we randomly create a lot of AOM samples of various position, shape and scale. (c) We evaluate each AOM by brightness orientation and silhouette of background subtraction. Only when moving object is in the block to use background subtraction information, we keep on searching for optimal AOM until the reliable Ω-shape is found. (d) Tracking system is initialized when Ω-shape is found. We fit average shape of MLM since we assume that detected human is standing. We store color information of each rectangles and use this information in likelihood evaluation of tracking. Only when moving object is in the block to use background subtraction information, we keep on searching for optimal AOM until the reliable Ω-shape is found in Fig.3(c). Our system initializes MLM position after Ω-shape detection to move on to the tracking step in Fig.3(d). The position and scale of initial MLM is estimated from detected Ω-shape. We fit average shape of MLM since we assume that detected human is standing. We store color information of each rectangles and use this information in likelihood evaluation of tracking. V. HUMAN TRACKING In this chapter, we propose a human tracking framework using integrated model of AOM and MLM, explained in Section 3. Searching the best parameters of the integrated model is performed by particle filtering (PF). PF algorithm is an iterative algorithm including prediction, update and resampling. This is a kind of Bayesian estimator that applies Monte Carlo framework. The current state is estimated as the expectation of likelihood distribution. Random sampling of PF allows the effective search in the state space. Our human shape model has 7dimensions and in addition there are 5 dimensions in image space [position:(x,y), velocity:(u,v), scale:s]. The state space has 12 dimensions in our tracking system. Dynamical Model: The parameters of state space are given by position, velocity, scale and parameters of AOM and MLM. We model the change of positional value in state vector by linear uniform motion model when the human is walking in images. And the change of postural value in state vector is complicating when the human is sitting or bending. Hence, we use random walk and linear uniform motion model as the state transition model in Eq.1 3. x t = (x t, y t, u t, v t, s t, a 1, a 2, a 3, a 4, m 1, m 2, m 3 ) (1) x t = Fx t 1 + n (3) where x t is a state vector, F is a state transition matrix and n N(0,Σ) is Gaussian noise. Likelihood evaluation: We evaluate various state vector by Formulas shown as below. Eq.5 indicates the sum of orientation difference between the brightness gradient of images and the normal vector on AOM. In the case of silhouette, Eq.6 calculates the sum of pixel values from a distance transform image. If an Ω-shape is fitting the silhouette of background subtraction, this score become high value. The complexity of background edges doesn t affect likelihood evaluation because of combination of Eq.5 and 6. We can track Ω-shape accurately without the influence of background edges. MLM adopts color histogram and Bhattacharrya coefficient to calculate likelihood in Eq.7. Color histograms are insusceptible to postural change. Bhattacharrya coefficient is known as the method for measuring a distance of between two histograms. This tracking system decides the position of state space after likelihood distribution updating. L(x t ) = C 1 S edge + C 2 S sil + C 3 S color (4) S edge = 1 d I (x, y) d AOM (x, y) ds (5) s(x,y) Ω S sil = 1 D(x, y)ds (6) s(x,y) Ω S color = pm (n)q m (n) (7) m=1 n=1 F = (2) where S edge and S sil are fitting likelihood of AOM computed from brightness gradient and silhouette of background subtraction. S color is likelihood of MLM computed from color information. Ω is set of points representing Ω-shape, L(x t ) is likelihood of a particle, s(x, y) is point of AOM, d I (x, y) is gradient orientation of input image, d AOM (x, y) is gradient orientation of AOM, D(x, y) is value of distance transform image, C 1, C 2, C 3 are weight coefficients. m is the number of body parts, n is the number of histogram bins, p(n) is 2480

Fig. 4. Low dimensional human shape model and likelihood evaluation. (a)partial occlusion (b)postural change Fig. 5. Samples of tracking result. (c)blur Fig. 6.

The model color histograms are calculated during tracker initialization (model appearance initialization).

5 shows the examples of our tracking result using Ω- shape and main parts model. Ω-shape tracking (edge-based tracking) doesn t work properly in the case of blurry image shown as Fig.5(c).

Detection Experiment We implement the previous detection method that classifiers are created by CoHOG feature descriptor and SVM.

[9] shows CoHOG is better than HOG [dalal, 2005], LRF [Gavrilla], Haar Wavelet [Dollar], Shapelet and M-HOG [Maji] in real scenes.

4 Fig. 4. Low dimensional human shape model and likelihood evaluation. (a)partial occlusion (b)postural change Fig. 5. Samples of tracking result. (c)blur Fig. 6. (a)previous method (b)proposed method Detection results of previous and proposed method. model color histogram and q(n) is a sample color histogram of MLM. The model color histograms are calculated during tracker initialization (model appearance initialization). Fitting only each model in images is difficult because of occlusion, illumination change and some other changes. Fig.5 shows the examples of our tracking result using Ω- shape and main parts model. Ω-shape tracking (edge-based tracking) doesn t work properly in the case of blurry image shown as Fig.5(c). In this case, main-parts tracking (colorbased tracking) is effective method to capture human position. Combination tracking can complement each other. A. Detection Experiment We implement the previous detection method that classifiers are created by CoHOG feature descriptor and SVM. This method is known as high-accuracy method under complex scenes. [9] shows CoHOG is better than HOG [dalal, 2005], LRF [Gavrilla], Haar Wavelet [Dollar], Shapelet and M-HOG [Maji] in real scenes. We compared the detection rate of our proposed method and previous method. Table I shows that the detection rate of proposed method is slightly lower than that of previous method. Moreover, our proposed method runs at 0.19 second per frame. In previous method, extracting high dimensional feature is required to detect human in each image. VI. EXPERIMENTS TABLE I. COMPARE DETECTION RATE AND PROCESSING TIME. In order to validate our proposed method, we have carried out following experiments. We used a conventional PC equipped with Intel Core i7 2.80GHz in these experiments. Detection rate Processing time Previous 88.3%(174/197) 7.28 sec / frame Proposed 80.2%(158/197) 0.19 sec / frame 2481

5 Additionally, you only obtain rectangle information with those methods. On the other hand, proposed method can evaluate with only simple features edge and silhouette. Our detection method provides the significant information for tracking step. Fig.6 shows the detection results of both detection methods. While previous method provides only the whole rectangle of the target, our method provides more detailed scale and shape information, namely contour, of the tracking target in each frame. B. Tracking Experiment We performed the two precision tests to show the effectiveness of our method. One is to show the robustness with occlusion. In this experiment, we used 23 occlusion scenes extracted from CAVIAR dataset [16]. Since this dataset is not included postural scenes, we performed the precision test on our own dataset to show the performance of our method with postural change. We assumed shopping scenes in our dataset. In shopping scenes, there are many postural changes such as bending and sitting due to selecting products. We can evaluate the robustness of our tracking method not only under occlusion but also under postural change. In each scene, we give human region information of initial frame to the tracking system. Table II shows our tracking is high accuracy in any scenes. Wu et al. used CAVIAR dataset to evaluate their tracking method in [7]. In the [7], persons are detected and tracked by separating into each body part. This is effective for partial occlusion and their method is state-of-the-art approach for human tracking. In our tracking method, persons are tracked by separating into each body part too. To show the table II and Fig.7, our tracking accuracy under occlusion is high as well as their tracking accuracy. Separating and evaluation of each body part is effective for partial occlusions. In addition, our approach can precisely track under postural change scenes shown in Fig.8. This experiment indicates our tracking is the notable method in the case of postural change not only occlusion. In previous method tries to capture the human as a rectangle region. This method cannot get the human region and its color histograms of each body part properly. Our approach can capture the body parts flexibly in the situation of postural change. MLM can represent the deformation such as shortening or rotation of body parts. These information are important for more detailed pose estimation or activity recognition. In this way, our method can be applied to not only tracking in occlusion and postural change scenes but also rough pose estimation. These tracking results are very important to recognize human activities in security system and commerce facilities. proposed method can extract rough pose and the position of shoulder joint as the result of separate body tracking and Ω- shape. This method is suitable for human activity recognition. We would like to realize the real-time and robust action recognition to improve each process as a future work. REFERENCES [1] Thomas B. Moeslund, Adrian Hilton, et al.: A survey of advances in vision-based human motion capture and analysis, Computer Vision and Image Understanding 2006, pp [2] D. Comaniciu, V. Ramesh, P. Meer: gkernel-based object trackingh, IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol.25, No.5, pp , [3] C. Yang, R. Duraiswami, D. Larry: Efficient mean-shift tracking via a new similarity measure, IEEE Conference on Computer Vision and Pattern Recognision, pp , [4] K. Nummiaro, E. Koller-Meier, L. Van Gool: An Adaptive Color-Based Particle Filter, Image and Vision Computing, vol.21, no.1, pp , [5] K. Okuma, A. Taleghani, N. De Freitas, et al: A boosted particle filter: Multitarget detection and tracking, COMPUTER VISION - ECCV2004, vol.3021, pp.28-39, [6] H. Palaio, J. Bastista: Multi-Object Tracking Using an Adaptive Transition Model Particle Filter with Region Covariance Data Association, International Conference on Pattern Recognition 2008, pp.1-4, [7] B. Wu, R. Nevatia: Detection and tracking of multiple, partially occluded humans by Bayesian combination of edgelet based part detectors, International Journal of Computer Vision, pp , [8] S. Kong, M.K. Bhuyan, C. Sanderson, Brian C. Lovell: Tracking of Persons for Video Surveillance of Unattended Environments, International Conference on Pattern Recognition 2008, pp.1-4, [9] Tomoki Watanabe, Satoshi Ito, Kentaro Yokoi: Co-occurrence Histograms of Oriented Gradients for Human Detection, IPSJ Transactions on Computer Vision and Applications, Vol.2, pp.39-47, 2010 [10] N. Dalal, B. Triggs: Histograms of Oriented Gradients for Human Detection, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol.1, pp , 2005 [11] P. Sabzmeydani, G. Mori: Detecting Pedestrians by Learning Shapelet Features, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.1-8, 2007 [12] S. Maji, A.C. Berg J. Malik: Classification using Intersection Kernel Support Vector Machine is Efficient, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2008 [13] M. Li, Z. Zhang, K. Huang, T. Tan: Estimation the number of people in crowded scenes by mid based foreground segmentation and headshoulder detection, ICPR2008, pp.1-4, 2008 [14] Min Li, Zhaoxiang Zhang, Kaiqi Huang, Tieniu Tan: RAPID AND ROBUST HUMAN DETECTION AND TRACKING BASED ON OMEGA-SHAPE FEATURES, International Conference on Image Processing 2009, pp , 2009 [15] Pedro F. Felzenszwalb, Daniel P. Huttenlocher: Pictorial Structures for Object Recognition, International Journal of Computer Vision 2005, pp.55-79, 2005 [16] TABLE II. PRECISION OF OUR TRACKING UNDER OCCLUSION AND POSTURAL CHANGE. Occlusion Postural change Total Processing Time Tracking rate 73.9%(17/23) 84.2%(16/19) 80.0% 412.3msec VII. CONCLUCIONS AND FUTURE WORK In this paper, we proposed a novel human tracking method using the integrated model combining Ω-shape head and shoulder model (AOM) and main parts model (MLM). This tracking is robust in the occlusion and postural change scenes. And our 2482

6 Powered by TCPDF ( Fig. 7. Tracking results of CAVIAR dataset. Fig. 8. Tracking results of our dataset. 2483

FAST HUMAN DETECTION USING TEMPLATE MATCHING FOR GRADIENT IMAGES AND ASC DESCRIPTORS BASED ON SUBTRACTION STEREO

FAST HUMAN DETECTION USING TEMPLATE MATCHING FOR GRADIENT IMAGES AND ASC DESCRIPTORS BASED ON SUBTRACTION STEREO Makoto Arie, Masatoshi Shibata, Kenji Terabayashi, Alessandro Moro and Kazunori Umeda Course