Random Subspace Method for Gait Recognition

Transcription

1 Random Subspace Method for Gait Recognition Yu Guan 1,Chang-Tsun Li 1 and Yongjian Hu 2 Department of Computer Science, University of Warwick, Coventry, UK {g.yu, c-t.li}@warwick.ac.uk 1, yongjian.hu@dcs.warwick.ac.uk 2 Abstract Overfitting is a common problem for gait recognition algorithms when gait sequences in gallery for training are acquired under a single walking condition. In this paper, we propose an approach based on the random subspace method (RSM) to address such overlearning problems. Initially, two-dimensional Principle Component Analysis (2DPCA) is adopted to obtain the full hypothesis space (i.e., eigenspace). Multiple inductive biases (i.e., subspaces) are constructed, each with the corresponding basis vectors randomly chosen from the initial eigenspace. This procedure can not only largely avoid overadaptation but also facilitate dimension reduction. The final classification is achieved by the decision committee which follows a majority voting criterion from the labeling results of all the subspaces. Experimental results on the benchmark USF HumanID gait database show that the proposed method is a feasible framework for gait recognition under unknown walking conditions. Keywords-overfitting avoidance; random subspace method; gait recognition; biometrics I. INTRODUCTION Compared with other biometrics like face or iris recognition, the most significant advantage for gait recognition is that it can be applied unobtrusively at a distance. According to the early medical and physiological studies, the human gait has 24 different components, which indicates that the gait pattern is unique for individuals [6]. However, covariate factors that can affect the recognition performance do exist. These include walking surface, shoe type, clothing, carrying condition, medical condition, emotions etc. How to address these problems is an acute challenge. There are two mainstreams in gait recognition, namely appearance-based gait recognition and model-based gait recognition. The former focuses on statistical analysis on the appearance of the gait video frames, whereas the latter is on estimating human body structure parameters. The average silhouette over one gait cycle, known as Gait Energy Image (GEI) is widely used in recent appearance-based gait recognition algorithms because of its simplicity and effectiveness [1][3][4][7][8][13]. Compared with the traditional frame-based methods (e.g., the baseline algorithm with direct frame matching [2]), the averaging operation makes GEI less sensitive to segmentation errors [4]. However, with gallery GEIs obtained under similar physical conditions, the learned features may overfit the training data. In this case, the performance may not be satisfactory when the overlearned model encounters a query GEI from an unknown walking condition. Fig. 1 provides the GEI examples to explain this challenge. From the classification viewpoint, it is a difficult task due to the small inter-class variations for training as illustrated in Fig. 1(a), and the large intra-class variations for testing as illustrated in Fig. 1(b)-(h). Overfitting may happen under such circumstances. For example, the features of individual shadows in Fig. 1(a) can be learned by the trained model, but in most cases, it is unreasonable if the classification is performed based on these features. To overcome overadaptation, cutting and fitting scheme is employed in [4] to create additional synthetic training samples by gradually cutting the silhouette shadow and resizing to simulate the walking surface distortion. The trained model shows its insensitiveness to different walking surfaces, but it is not robust against other covariate factors like carrying condition. In [7], the Gabor kernel is applied to the GEIs with respect to different orientations and different scales. Competitive performance is achieved under the influence of different shoe types, camera viewpoints and carrying conditions, but it is not invariant to different walking surfaces and time. Combining the merits of both methods, Huang et al. [3] expand the Gabor-feature-based gallery to cover more variations by interpolating the imagespecific Gaussian Mixture Models. Although promising average recognition rate is reported, for some unknown walking conditions not fully covered in the virtual sample construction process, the recognition rate is extremely low (e.g., the recognition accuracy is only 3% under the combined influence of shoe type, clothing, and elapsed time). As the physical conditions for a probe gait sequence is unknown, it is very challenging to create the right synthetic training samples for robust gait recognition. The random subspace method (RSM), as an ensemble construction for classification, offers an alternative way to solve the dilemma of accuracy optimization and overfitting [5]. The idea behind RSM is, instead of using a single feature space for classification, multiple subspaces are constructed from the initial eigenspace in a random manner. Each random subspace can be deemed as an inductive bias and there is a classifier for labeling. The final classification is determined by the decision committee which combines all the labeling results associated with the corresponding subspaces. Although the mislabeling problem may exist for a single random subspace, the performance of their combination can be much better, especially when the dataset has a large number of features and not too few samples [5].

2 The remainder of this paper is organized as follows: Section II describes the details of the proposed approach. Experimental results and analysis are provided in Section III, and Section IV concludes the paper. (a) (b) (c) (d) (e) (f) (g) (h) Figure 1. GEI examples from the USF gait database for individual recognition. (a) The training samples of different individuals walking in the same walking condition, (b)-(h) The query GEIs from one individual under the influence of the following covariate factors: (b) viewpoint, (c) walking surface, (d) viewpoint and walking surface, (e) carrying condition, (f) carrying condition and viewpoint, (g) elapsed time, shoe type, and clothing, (h) elapsed time, shoe type, clothing and walking surface. In this work, to avoid overfitting, rather than creating synthetic samples in the gallery by simulating distortion or adding more features by applying certain functions to the original data, we utilize RSM for robust gait recognition. First, two-dimensional Principle Component Analysis (2DPCA) is employed for the initial eigenspace construction and the highly redundant GEIs can be projected into a smaller number of reconstruction coefficients in an uncorrelated eigenspace. Considering in most cases that GEIs in gallery for training are acquired in a single walking condition with small inter-class variations, directly using 2DPCA as a dimension reduction method by discarding lowenergy components cannot well address the overlearning problem. The coefficients corresponding to high-energy eigenvectors in eigenspace can represent the original data well, but they are not necessarily discriminative. To make use of discriminative information for robust classification, we combine weak classifiers to avoid overadaptation. Multiple inductive biases are constructed, with the corresponding basis vectors randomly chosen from the initial eigenspace. Each GEI is represented by different sets of reconstruction coefficients associated with different random subspaces. For a certain random subspace, we further project the corresponding reconstruction coefficients into the canonical space with optimal class separability. The nearest mean classifier is then employed for labeling for each random subspace and the final classification is achieved by majority voting from all the weak classifiers. The proposed approach has several desirable properties: It is easy to comprehend and implement. By setting up multiple inductive biases, it effectively avoids overfitting the inadequate training set, which is normally obtained under a single walking condition. Even based on the simplest input features (i.e., pixel intensities), very competitive performance can be achieved on the benchmark USF gait database. It is a feasible framework for gait recognition and can be further extended. II. PROPOSED APPROACH Random subspace ensembles have been successfully applied in face recognition [11][12]. It is summarized in [11] that the ensemble techniques can usually increase the generalization for the trained model, which can combat overfitting. Motivated by [5][11][12], this work explores such techniques in the context of gait recognition, which is more challenging as it involves the problems of larger intraclass variations and smaller inter-class variations. A. Combining RSM and 2DPCA PCA is usually employed as a preprocessing tool for the RSM. The purpose of PCA is to obtain a number of principle components to represent the original data for dimension reduction or data decorrelation. However, for high dimensional data, PCA is usually computationally expensive and sometimes difficult to evaluate the covariance matrix accurately [10]. 2DPCA, based on 2D matrices rather than 1D vectors, can address these issues [10] and this work exploits 2DPCA as our initial eigenspace (i.e., full hypothesis space) construction. Assume we have n sample GEIs I i (i=1,,n) in gallery, and first we obtain the mean M of the GEIs: 1 n M = I. (1) i= 1 i n The covariance/scatter matrix S can then be estimated: 1 n T S = ( I M) ( I M). (2) j= 1 n j Based on the scatter matrix S, the eigenvectors can be computed for the purpose of decorrelation. Eigenvectors with zero eigenvalues are removed, and the rest are retained as candidates for the random subspaces construction. L random subspaces R 1,,R L can be generated by randomly selecting N eigenvectors from the candidates. Each GEI can be projected into the L subspaces with the corresponding L sets of reconstruction coefficients. After the 2DPCA projection for each random subspace, one GEI can be represented as L sets of coefficients corresponding to L N-dimensional random subspaces, which can be used as features for further processing. B. Linear Discriminant Analysis To achieve optimal class separability, Linear Discriminant Analysis (LDA) is adopted to further project the coefficients into the canonical space which maximizes the Fisher s criterion. For each random subspace R k (k=1,,l), there is a transition matrix W k that maximizes the ratio of the between-class scatter matrix S k B to the withinclass scatter matrix S k W: T k W S k B W J ( W ) =. (3) T k W S W W j

3 However, to meet the input requirement of the traditional LDA, data have to be vectorized which may lead to high dimensionality. To avoid high dimensionality which may cause expensive computational cost and sometimes the singularity problem, in this work, the matrix-based 2DLDA is employed. C. Classification For the k th subspace out of L random subspaces, we assume W k, R k are corresponding canonical space transition matrix and eigenspace transition matrix. A gait sequence with n GEIs I i (i=1,,n) can be projected into n features matrices Y k i : k k k Yi = W IiR. (i = 1,...,n) (4) Assume there are c classes of gait sequences in gallery, each with n j (j=1,,c) feature matrices, so each class can be k represented by the centroid G j (j=1,,c) of its corresponding feature matrices. Similar to the set-to-set distance defined in [4], for a probe gait sequence P k k with n p feature matrices Y l (l=1,,n p ), the dissimilarity between P k and a certain class G k j can be measured by the average of the distances between each feature matrix and G k j,i.e., k k 1 n (, ) p k k DP G = Y G. (5) j l = 1 l j n p D() defines the set-to-set distance for two gait sequences and the gait sequence P k is labeled as G k m if k k c k k DP (, Gm ) = min j= 1 DP (, Gj ). (6) As there are L subspaces, the final classification is achieved by majority voting from the L labeling results. III. EXPERIMENTS A. Data and Parameters The University of South Florida (USF) gait database is the largest publicly available human gait database with 1870 gait sequences from 122 individuals. Five main covariates are considered: camera viewpoints (left/right), two different shoe types, surface types (grass/concrete), carrying conditions (with/without a briefcase), and elapsed time between samples being compared (May/November). There are 12 predesigned experiments for the purpose of testing a single covariate or a combination of covariates, with the size of the probe set ranging from 33 to 122, as shown in TABLE I. Sarkar et al. [2] also extracted the binary gait silhouettes as well as GEIs, which can be downloaded at In this work, we perform our method directly on the GEIs from the USF database. To evaluate the performance of the algorithms, we adopt the rank-1/rank-5 average recognition accuracy. Rank-1 (resp. rank-5) shows the correct individual is ranked as the top 1 candidate (resp. top 5 candidates), whereas the average recognition accuracy is the average recognition rate over all probes in terms of rank-1/rank-5. Figure 2. Performance distribution with respect to the dimension of the random subspace N. There are two main parameters in our approach: the dimension number of the random subspace N and the number of random subspaces L (i.e., the number of weak classifiers). Intuitively, if the value of N is too small / too large, each weak classifier for a certain random subspace will face the underlearning/overlearning problem, so a suitable N is essential for the performance. However, considering there is no suitable validation set in gallery for parameter estimation, so in this paper, like most of the prior works [1][3], we empirically choose the parameters. It was verified in [5] that the recognition accuracy does not decrease with respect to the increasing number of random subspaces. Based on this assumption, we find the optimal random subspace dimension N by fixing the value of L. When we set L=50, the average rank-1 performance distribution shows that the optimal N should be at the range of [11, 20], which can be seen from Fig. 2. As the aim of this paper is to explore the effectiveness of the RSM in the context of gait recognition, rather than achieving a higher performance through intensive parameter calibration, we simply use the median value N=16 for the rest of our experiments, with L=50, L=100, L=300, L=500, and L=1000, respectively. TABLE II illustrates the performance when L=50 for the 12 predesigned experiments, with each run 10 times. For a certain experiment, due to random nature, results for each run vary and for fair comparison with other methods, we run each experiment 10 times and report the mean recognition rate as performance. More detailed results can be seen in TABLE III, when different values of L are employed. B. Performance Evaluation The experimental results as well as comparison with other classical methods are shown in TABLE III. As we expected, the performance is improved with the increasing number of random subspaces as shown in Fig. 3. There is a tradeoff between the performance and the computational cost. It is worth mentioning that even with a small value of L

4 TABLE I. 12 PREDESIGNED EXPERIMENTS FOR GAIT RECOGNITION IN THE USF DATABASE. Experiment A B C D E F G H I J K L Probe Size Covariates V H VH S SH SV SHV B BH BV THC STHC (Legends: V-View, H-Shoe, S-Surface, B-Briefcase, T-Time, AND C-Clothing) TABLE II. RANK-1 PERFORMANCE OF 12 EXPERIMENTS, WITH EACH RUN 10 TIMES (L=50, N=16). Probe Set Mean (%) Standard Deviation Maximum (%) Minimum (%) A B C D E F G H I J K L (e.g., L=100), our method still outperforms most of the others, which means our method is also feasible for real time applications. With the L=1000, the proposed approach achieves the best result in terms of average rank-1 and outperforms all the other algorithms except DNGR [9] in terms of average rank- 5 recognition rate. Note that DNGR uses additional manually cleaned silhouettes for training, which are not publicly available. The result of the proposed approach demonstrates its robustness for the following covariates: camera viewpoint, shoe type, carrying condition and their combinations (i.e., probe sets A-C, H-J). For experiments under the main influence of the combination of elapsed time and clothing (i.e., probe sets K-L), although our method still outperforms most of the others, the recognition rate is not satisfactory for practical applications. Up to now, covariate factors combining elapsed time and clothing remain an open question for all the gait recognition algorithms. Apart from DNGR [9] which involves manual adjustment, prior works [1][3][4] report better results in experiments with the walking surface covariate (i.e., probe sets D-G). Gallery expansion by specific distortion-simulation [4] or interpolation and extrapolation [3] shows the insensitiveness to different walking surfaces to some extent. However, for the covariate variations that have not been successfully covered (e.g., carrying condition or clothing), the performance can be extremely low. Considering the fact that the walking surface covariate may rotate human silhouette in GEIs to some degree, Image-to-Class distance is utilized in [1] by allowing feature matching to be carried out within a spatial neighborhood. Although this enhances the performance in experiments D-F, it is not generally robust against other covariates and is computationally expensive. Figure 3. Performance distribution with respect to the number of random subspaces L. Although our method only improves 0.2% against PDF- RTRDA [3], it should be pointed out that we only use baseline input features (i.e., pixel intensities). In [3], for effectiveness feature evaluation, Huang et al. also conducts RTRDA classification method based on pixel intensities (referred as to Gray-RTRDA), and as can be seen from TABLE III, recognition accuracy is much enhanced (e.g., by 20.03% in terms of rank-1) by employing effective local Gabor-PDF features (i.e., PDF-RTRDA). This indicates the significant potential of our RSM framework. For example, the holistic method based on pixel intensities used in this paper is sensitive to spatial misalignment (which happens with the covariate walking surface), and the performance can be further improved if we adopt more sophisticated features (e.g., local Gabor-PDF features) under this framework in the future. C. Overfitting Avoidance When gait sequences for training are all under the same walking condition, it is challenging to select the discriminative features. Features with the large energy in the training set do not necessarily mean that they are also discriminative. For example, individual shadows in gallery should not be learned for classification in most cases, as mentioned in Section I. However, for a query GEI under the same walking surface as the training samples, the shadows are contributing positively to recognition, as verified by Liu et al. [13]. Besides, the intra-class variances can be very large (e.g. elapsed time or clothing), which may decrease the discriminative power of the features in the trained model. A classification task only with a pre-trained feature set is challenging when the walking condition for a query GEI is unknown and unpredictable.

5 TABLE III. RANK-1/RANK-5 PERFORMANCE (%) COMPARISION ON THE USF DATABASE Probe Set A B C D E F G H I J K L Average Probe Size RANK-1 Baseline [2] GEI real [4] GEI Fusion [4] DNGR [9] MMFA [8] GTDA [7] Image-to-Class [1] Gray-RTRDA [3] PDF-RTRDA [3] DPCA RSM-2DPCA L= RSM-2DPCA L= RSM-2DPCA L= RSM-2DPCA L= RSM-2DPCA L= RANK-5 Baseline [2] GEI real [4] GEI Fusion [4] DNGR [9] MMFA [8] GTDA [7] Image-to-Class [1] Gray-RTRDA [3] PDF-RTRDA [3] DPCA RSM-2DPCA L= RSM-2DPCA L= RSM-2DPCA L= RSM-2DPCA L= RSM-2DPCA L= Covariates such as camera viewpoint or shoe type, have little impact on the query GEI qualities (e.g., Fig. 1(b)) and most of the algorithms perform relatively well in the corresponding experiments A-C, as can be seen from TABLE III. For experiments K-L, in which the probe sets look significantly different from the training samples (e.g., Fig. 1(g)-(h)), the performance of each algorithm is relatively low. All the algorithms listed in TABLE III, including the proposed one seem overfitted. However, compared with other algorithms, the proposed RSM-based approach performs classification through voting, based on different inductive biases. It not only achieves the best general performance, but also has at least moderate recognition accuracy for each experiment, which indicates the generalization of our approach. To illustrate that our work can avoid overfitting to a large extent, we also perform the classical 2DPCA algorithm on the 12 experiments for comparison, with parameters set to the same values as in the proposed work. Classical 2DPCA keeps the eigenvectors corresponding to N largest eigenvalues to project a feature set. Similarly, 2DLDA is also applied for optimal class separability. As shown in TABLE III, without RSM framework, the average recognition rate of the classical 2DPCA is only slightly better than the baseline algorithm [2]. When it combines with the RSM, the performance is significantly enhanced from an average rank-1(resp. rank-5) 45.24% to 65.66% (resp % to 81.18%), which proves our proposed RSM-based approach can effectively address the overlearning issue. IV. CONCLUSION In this paper, we present a RSM-based approach for gait recognition under various unknown walking conditions. Given the fact that the gait training set is normally expensive to establish and inadequate, whereas the walking condition of a query gait is unpredictable, gait recognition involves large intra-subject variations and small inter-subject variations and overfitting avoidance is crucial. Most of the prior works attempt to train the model with the minimal reconstruction error criterion in the first place. However, reconstruction is not a sufficient and necessary condition for discrimination, and perfect reconstruction is more prone to overadaptation in the context of gait recognition. In this work, we set up multiple inductive biases for the model generalization. To our best knowledge, such an approach has never been attempted in the area of gait recognition. Compared with other classical algorithms, the proposed approach achieves a better performance and is relatively robust to different walking conditions. Based on this work, to further enhance the performance, there are some aspects worth exploring:

6 1.) Effective input features, e.g. the local features or patch-based features. 2.) New hypothesis space construction, instead of eigenspace based on 2DPCA. 3.) New fusion rules for classification decision, rather than being dependent solely on majority voting. REFERENCES [1] Y. Huang, D. Xu, and T. Cham, "Face and Human Gait Recognition Using Image-to-Class Distance," IEEE Transactions on Circuits and Systems for Video Technology, vol.20, no.3, pp , March [2] S. Sarkar, P.J. Phillips, Z. Liu, I.R. Vega, P. Grother, and K.W. Bowyer, "The humanid gait challenge problem: data sets, performance, and analysis," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.27, no.2, pp , Feb [3] Y. Huang, D. Xu, and F. Nie, "Regularized Trace Ratio Discriminant Analysis with Patch Distribution Feature for human gait recognition," IEEE International Conference on Image Processing (ICIP), pp , Sept [4] J. Han, and B. Bhanu, "Individual recognition using gait energy image," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.2, pp , Feb [5] T. K. Ho, "The random subspace method for constructing decision forests," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.20, no.8, pp , Aug [6] M.P. Murry Gait as a Total Pattern of Movement, American Journal of Physical Medicine, vol.46, pp , [7] D. Tao, X. Li, X. Wu, and S.J. Maybank, "General Tensor Discriminant Analysis and Gabor Features for Gait Recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.29, no.10, pp , Oct [8] D. Xu, S. Yan, D. Tao, S. Lin, and H. Zhang, "Marginal Fisher Analysis and Its Variants for Human Gait Recognition and Content- Based Image Retrieval," IEEE Transactions on Image Processing, vol.16, no.11, pp , Nov [9] Z. Liu, and S. Sarkar, "Improved gait recognition by gait dynamics normalization," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.6, pp , June [10] J. Yang, D. Zhang, A.F. Frangi, and J. Yang, "Two-dimensional PCA: a new approach to appearance- based face representation and recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.26, no.1, pp , Jan [11] N.V.Chawla, and K.W. Bowyer, "Random subspaces and subsampling for 2-D face recognition," IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR), vol.2, pp , June [12] X. Wang, and X. Tang, "Random sampling LDA for face recognition," IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol.2, pp , July [13] Z. Liu, and S. Sarkar, "Simplest representation yet for gait recognition: averaged silhouette," International Conference on Pattern Recognition(ICPR), vol.4, pp , Aug 2004.