Towards Understanding the Limits of Gait Recognition

Transcription

1 Towards Understanding the Limits of Gait Recognition Zongyi Liu, Laura Malave, Adebola Osuntugun, Preksha Sudhakar, and Sudeep Sarkar Computer Vision and Image Informatics Laboratory Computer Science and Engineering University of South Florida Tampa, FL ABSTRACT Most state of the art video-based gait recognition algorithms start from binary silhouettes. These silhouettes, defined as foreground regions, are usually detected by background subtraction methods, which results in holes or missed parts due to similarity of foreground and background color, and boundary errors due to video compression artifacts. Errors in low-level representation make it hard to understand the effect of certain conditions, such as surface and time, on gait recognition. In this paper, we present a part-level, manual silhouette database consisting of 71 subjects, over one gait cycle, with differences in surface, shoe-type, carrying condition, and time. We have a total of about 11, manual silhouette frames. The purpose of this manual silhouette database is twofold. First, this is a resource that we make available at for use by the gait community to test and design better silhouette detection algorithms. These silhouettes can also be used to learn gait dynamics. Second, using the baseline gait recognition algorithm, which was specified along with the HumanID Gait Challenge problem, we show that performance from manual silhouettes is similar and only sometimes better than that from automated silhouettes detected by statistical background subtraction. Low performances when comparing sequences with differences in walking surfaces and time-variation are not fully explained by silhouette quality. We also study the recognition power in each body part and show that recognition based on just the legs is equal to that from the whole silhouette. There is also significant recognition power in the head and torso shape. Keywords: Gait recognition, gait analysis, segmentation, ground truth, silhouettes, human shape, gait dynamics, pedestrian detection, behavioral biometrics 1. INTRODUCTION In the 197 s Cutting and Kozlowski, 1 using point light display based experiments patterned after those of Johansson s, 2 demonstrated the ability to recognize friends from gait. In computer vision, while gait analysis has been a research topic for a while, it is only recently that identification from gait has received attention and has become an active area of computer vision For biometics research, gait is usually refered to in its broadest sense to include both body shape and dynamics, i.e. any information that can be extracted from the video of a walking person to robustly identify the person under various condition variations. It is one biometric source that can be acquired at a distance, making it important for some early warning or monitoring applications that needs to perform recognition when the subject is far away. While one cannot expect it to be a perfect biometric source, it important to understand its scope. For some application scenarios, less than perfect recognition is better than nothing. However, before deploying gait base recognition systems, it is important to study the associated scientific questions such as: With what confidence level can we identify persons from their gait? Is gait good for verification or is it better for identification scenarios? What are the limitations of recognition from gait? What factors effect gait recognition and to what extent? Answers to these questions are not known. To help answer these kinds of questions, the HumanID Gait Challenge Problem was formulated. 16,17 It consists of (i) a large gait database with 187 sequences from 122 subjects, spanning 5 covariates: across view (about 3 degrees), across shoe-type, across briefcase, across surface types (grass/concrete), and across time; (ii) a set of twelve experiments to investigate the effect of five factors on performance. The five factors are studied both individually and in combinations. The results from the twelve experiments provide an ordering of the difficulty of the experiments. And, it consists of (iii) a baseline algorithm based on the Tanimoto distance between silhouettes, extracted by simple background differencing, to provide the needed performance benchmark. Some sample frames are shown in Figure 1. Send correspondences to Sudeep Sarkar, sarkar@csee.usf.edu, Telephone:

2 Table 1. Summary the top rank recognition for experiments A (viewpoint variation between probe and gallery), B (shoe-type variation), D (surface variation), and K (Time variation) in the Gait Challenge dataset. The numbers for the first two columns are as read from graphs in the cited papers. Exp. Fusion (UMD) 19 DTW (UMD) HMM (UMD) 21 Body Shape (CMU) 22 HMM (MIT) 23 Body (CAS) 24 Baseline (USF) 16 A (view) 52% 78% 99% 87% 88% 7% 79% B (shoe) 4% 65% 89% 81% 75% 59% 66% D (surface) % 29% 36% 21% 25% 34% 29% K (time) 3% # subjects in gallery For some of the key gait challenge experiments, Table 1 lists summary performances that have been so far reported in the literature by various gait recognition strategies, such those based on Hidden Markov Models (HMMs), Dynamic Time Warping (DTW), body shape matching, and shape moments. The listed performance numbers are the correct identification rates at the top most rank. This is a standard performance metric used in biometrics 18 for the identification scenario, where one is interested to find a match to a given probe from whole the gallery set, i.e. one-to-many match. (For the verification scenario, where one is interested in matching one probe to one gallery (one-to-one match), the performance is specified in terms of standard false alarm and detection rates. In general, identification is considered to be a harder problem than verification.) We see that performance of the baseline gait recognition algorithm is quite effective and competitive with other algorithms. Another observation of particular interest, is the significant effect of change in surface type on the identification rate; this effect is consistent across different types of gait recognition algorithms. For the baseline algorithm, we also see the significant effect of time variation of about six months. This effect of time on gait recognition has been documented by others, but on different data sets taken indoors. When the difference in time between gallery (the prestored template) and probe (the input data) is in the order of minutes, the identification performance ranges from 91% to 95%, 9, 13, 15 whereas the performances drop to 3% to 45% when the differences are in the order of months and days 11, 13, 19 for similar sized datasets. Since all the algorithms rely on silhouette as the low-level representation of choice, it is reasonable to suspect the flaky low-level processes for the low performances across surfaces and time. These experiments involve either a change in the background or substantial change in illumination, which very likely impact silhouette quality. Also, the high recognition rates that we see for the other covariates, such as view or shoe type, might be due to error correlation in the silhouettes, such as those due to shadows or holes. The error correlations is expected to be high since they involve data collected over a short time period of each other. To unravel these factors, we present results with part-level manual silhouettes. One conclusion of this work is that the quality of automated silhouettes, detected using standard techniques, involving background difference with Mahanalobis distance in the color space, does not seem to be the limiting factor in gait recognition. We arrive at this conclusion based on results with manually specified silhouettes. Second conclusion is that the recognition from legs is almost equal to that from the whole silhouette. The recognition from torso shape is also significant. 2. CREATION OF MANUAL SILHOUETTES For the sake of gaining insight into the relationship between recognition and silhouette quality, we manually created silhouettes over one gait cycle for 71 subjects under 4 different conditions, exhibiting the impact of cross shoe-type, surface, and time, respectively. This cycle was chosen to begin at the right heel strike phase of the walking cycle through to the next right heel strike, thus including one complete walking cycle. We attempted to pick this gait cycle from the same 3D location in each sequence, whenever possible. In addition, we tried to exclude the portion that included the calibration box with high contrast (see Figure 1), which frequently leads to high background subtraction errors. We not only mark a pixel as being from the background or subject, but provided more detailed specifications in terms of body parts too. We explicitly labeled the head, torso, left arm, right arm, left upper leg, left lower leg, right upper leg,

3 (a) (b) (c) (d) Figure 1. Sample frames in the gait challenge dataset as viewed from (a) the left camera on concrete surface, (b) the right camera on concrete surface, (c) the left camera on grass surface, (d) the right camera on grass surface. and right lower leg using different colors. Figure 2 shows some examples of part-level manual silhouettes corresponding to the original color images in the left column. Quality control checks looked for miscolored parts and backgrounds, randomly colored isolated pixels, errors on the boundary of the body, and missed body parts. Some of the difficulties encountered during the creating process include low-image quality due to varying overall intensity, occlusion of feet in the grass sequences, similarity of dark skin tones of some subjects with the background, frequent occlusion of the right arm, and the presence of dark or baggy clothing, which made it hard to delineate various body parts. However, despite these difficulties we were able to create pretty consistent quality silhouettes, as judged visually by another subject, across the subjects. 3. GAIT SIMILARITY COMPUTATION For recognition, the manual silhouettes are first height scaled so that they are all 128 pixels tall, which is around the average original resolution. Figure 3 shows images from the same person before and after scaling. In addition, the silhouettes are also centered to the middle of the frames so that frames can be compared by simple projection. Note that these simple operations, in essence, helps us in arriving at Kendall s notion of pre-shape by removing translation and scaling. Since all the subjects are upright and the camera is stationary, there is no need for rotation normalization before silhouette shape matching Multiple Gait Cycle Sequences We compare sequences, each with multiple but not necessarily equal gait cycles, using the baseline gait recognition algorithm that was specified along with the Gait Challenge problem. 17 The similarity computation is based on spatio-temporal correlation. Let the probe, consisting of M frames, and the gallery, consisting of N frames, be denoted by S P = {S P (1),, S P (M)} and S G = {S G (1),,S G (N)}, respectively. First, the probe (input) sequence is partitioned into subsequences,

4 Figure 2. Part level manual silhouettes over one gait cycle along with the corresponding color images, cropped around the person.

5 Figure 3. Top row shows the color images, cropped around the person for four different camera views. The middle row shows the corresponding part-level, manually specified silhouettes. And the bottom row shows the scaled silhouettes of the kind used by gait recognition algorithms. each roughly over one gait period, N Gait. Gait periodicity is estimated based on periodic variation of the count the number of foreground pixels in the lower part of the silhouette in each frame over time. This number will reach a maximum when the two legs are farthest apart (full stride stance) and drop to a minimum when the legs overlap (heels together stance). Second, each of these probe subsequences, S Pk = {S P (k),,s P (k + N Gait )}, is cross correlated with the given gallery sequence, S G. N Gait Corr(S Pk,S G )(l)= j=1 S(S P (k + j),s G (l + j)) (1) where, the similarity between two image frames, S(S P (i),s G ( j)), is defined to be the Tanimoto similarity between the silhouettes, i.e. the ratio of the number of common pixels to the number of pixels in their union. The overall similarity measure is chosen to be the median value of the maximum correlation of the gallery sequence with each of these probe subsequences. The strategy for breaking up the probe sequence into subsequences allows us to address the case when we have segmentation errors in some contiguous sets of frames due to some background subtraction artifact or due to localized motion in the background. ) Sim(S P,S G )=Median k (maxcorr(s Pk,S G )(l) (2) l The above strategy is effective for gait recognition when compared with performances of more complicated strategies, as we can see in Table 1. Note that both body shape and dynamics contribute to the similarity measure.

6 Figure 4. The illustration of the warping between sequences of different sizes Single Gait Cycle Sequences The baseline similarity computation strategy outlined above will not work, as is, for computing similarity from manual silhouettes, since they are defined over only one gait cycle. The correlation part is not needed. We simply linearly timewarp the sequences, but with the same frame-to-frame Tanimoto similarity measure. Let the two silhouette sequences be denoted by S 1 = {S 1 (1),, S 1 (M)} and S 2 = {S 2 (1),,S 2 (N)}. Without loss of generality, let M N. Then the similarity between the two sequences is Sim 1 (S 1,S 2 )= N i=1 ( S S 1 ( im ) N ),S 2(i) where, the similarity between two image frames, S(S 1 (i),s 2 ( j)), is defined to be the Tanimoto similarity between the silhouettes, i.e. the ratio of the number of common pixels to the number of pixels in their union. The overall similarity measure is the warped distance between the two sequences. Figure 4 illustrates this warping. Note that since the starting and ending stances of the two sequences are guaranteed to match, the chosen warping strategy is reasonable. 4. RECOGNITION RESULTS We compare the recognition from manual silhouettes with recognition based on automated silhouettes, both over one gait cycle and multiple gait cycles. The comparison over one gait cycle is, of course, the fairer of the two. We match each probe sequence to the gallery sequences, thus obtaining a similarity matrix with size that is the number of probe sequences by the gallery size. Following the pattern of the FERET evaluations, 18 we measure performance for both identification and verification scenarios using cumulative match characteristics (CMCs) and receiver operating characteristics (ROCs), respectively. In the identification scenario, the task is to identify a given probe to be one of the given gallery images. To quantify performance, we sort the gallery images based on computed similarities with the given probe. In terms of the similarity matrix, this would correspond to sorting the rows of the similarity matrix. If the correct gallery image corresponding to the given probe occurs within rank k in this sorted set, then we have a successful identification at rank k. A cumulative match characteristic plots these identification rates (P I ) against the rank k. In the verification scenario, a (3)

7 Table 2. The summary of the identification rate (P I ) at rank 1 and verification rate (P V ) at 1% false alarm rate (P F ) of the automated silhouettes and manual silhouettes. Exp. Automated Multi-cycle P I at 1 Automated Single-cycle Manual Single-cycle Automated Multi-cycle P V at 1% P F Automated Single-cycle Manual Single-cycle B (Shoe-type) 81% 54% 49% 83% 59% 39% D (Surface) 39% 24% % 46% % 21% H (Briefcase) 78% 37% 12% 73% 39% % K (Time) 15% 9% 12% 15% 9% 9% (a) (b) 9 9 Manual Single cycle (c) (d) Figure 5. The cummulative match characteristics capturing identification performance from the automated multi-cycle, automated single-cycle, and manual single-cycle silhouettes for variation in (a) shoe-type, (b) surface, (c) briefcase, and (d) time between probe and gallery. system either rejects or accepts if a person is who they claim to be. Operationally, a person presents (1). a new signature, the probe, and (2). an identity claim. The system then compares the probe with the stored gallery sequence that corresponds to the claimed identity. The claim is accepted if the match between the probe and gallery is above an operating threshold, otherwise it is rejected. For a given operating threshold, there is a verification rate (or detection rate) and a false accept rate. Changing the operating threshold can change the verification and false accept rates. The complete set of verification and false accept rates is plotted on a receiver operating characteristic (ROC).

8 Verification Rate 5 4 Verification Rate False Alarm Rate (a) False Alarm Rate (b) 9 9 Manual Single cycle Verification Rate 5 4 Verification Rate False Alarm Rate (c) False Alarm Rate (d) Figure 6. The receiver operating characteristics capturing verification performance from the automated multi-cycle, automated singlecycle, and manual single-cycle silhouettes for variation in (a) shoe-type, (b) surface, (c) briefcase, and (d) time between probe and gallery. First, we consider performance based on the whole silhouette. Figures 5 and 6 plot the Cumulative Match Curves (CMCs) for the first 5 rank and the Receiver Operating Characteristics (ROCs) up to the 5% false alarm rate, respectively. The results are summarized in Table 2. We see that the identification rate at rank 1 of the automated multi-cycle silhouettes is 81%, 39%, 78%, and 15% for cross shoe-type, surface, briefcase, time, respectively. This rates are, of course, higher than that obtained with just single cycle gait sequences. For the automated single cycle silhouettes, the number is 54%, 24%, 37%, and 9%, respectively. And for the manual single cycle silhouettes, the number is 49%, %, 12%, and 12%, respectively. The non-parametric Mcnemar s test confirms that the differences of the identification rates are statistically significant, except for the observed differences for the experiment across time. Comparing the single cycles performances, we see that the recognition with manual silhouettes outperforms the manual silhouette performance for all the experiments except for the time difference experiment. The biggest improvement is for the experiment comparing sequences with briefcase with sequences without briefcase. The removal of the briefcases in the manual silhouettes possibly contributed to higher recognition rates. Another observation worth noting is that performances when comparing sequences with differences in walking surface condition and time is still low with manual silhouettes, suggesting that silhouette quality does not fully explain the low performances. Second, we consider recognition from parts. What are the relative recognition rates based on silhouette portions from individual body parts, i.e. legs, hands, torso, or their combination? The manual silhouettes, which are specified at part-level, readily facilitates such as study. Figure 7 shows the CMCs upto rank 5 and Table 3 summarizes the top

9 Arms Only Arms + Legs Head + Torso Legs Only Full Body 9 8 (a) Arms Only Arms + Legs Head + Torso Legs Only Full Body 9 8 (b) Arms Only Arms + Legs Head + Torso Legs Full Body Arms Only Arms + Legs Head + Torso Legs Only Full Body (c) (d) Figure 7. The cummulative match characteristics capturing identification performance from body parts in the manual silhouettes for variations in (a) shoe-type, (b) surface, (c) briefcase, and (d) time between probe and gallery. Table 3. The top rank recognition rates and verification rates at 1% false alarm rate from different body parts for the key experiments. P I at rank 1 P V at P F is 1% B (View) D (Surface) H (Carry) K (Time) B (View) D (Surface) H (Carry) K (Time) Hands 22% % 8% 6% 24% 15% 12% 6% Hands+Legs 54% 24% 16% 12% 37% 18% 16% 9% Head+Torso 37% 23% 25% 12% 42% 23% 18% 9% Legs 49% 15% 16% 12% 51% 18% 16% 12% Full Body 49% % 12% 12% 39% 21% % 12% rank identification rates and the verification rate for a 1% false alarm rate, when considering different body portions for the different experiments. We see that the recognition from just the legs match the recognition from the full body. The recognition from the combination of legs and hands, which convey the dynamic component of gait, somewhat outperforms the recognition from full body silhouettes. Also notice that significant recognition power also exists in the head and torso portions, which basically conveys body shape.

10 5. CONCLUSION We presented a large manual silhouettes database for gait recognition and modeling research. The database contains part-level silhouettes, over one gait cycle, for about 7 subjects, spanning variation in surface, shoe, carrying condition, and time. This database could be used by others for a variety of purposes, such as designing better silhouette detection algorithms, learning gait kinematic models, and studying human gait variations. It is marker-less, normal, non-threadmill, gait data that is essentially uncorrupted by placement of markers and other intrusive or restrictive devices. It has been noted by many researchers that the use of markers or threadmill changes gait. Gait recognition experiments with the manual silhouettes suggest several interesting conclusions. First, recognition with manual silhouettes improved over automated silhouettes. However, while the improvement was the largest for comparing sequences with and without briefcase, it was not large for surface and time variation experiments. This suggests that automated silhouette quality is only partly to explain the low performance on these experiments. Second, recognition from just silhouette portions from the legs is almost equal to that from the whole body. The head and torso portions, which capture the static body shape, also has recognition power. This seems to suggest that both gait dynamics and body shape contribute towards recognition from gait video. 6. ACKNOWLEDGMENT This research was supported by funds from DARPA (AFOSR-F ) and the USF College of Engineering REU program. Thanks to the gait researchers at CMU, MIT, and Southampton to help correct some of the errors in the manual silhouettes. REFERENCES 1. J. E. Cutting and L. T. Kozlowski, Recognition of friends by their walk, Bulletin of the Psychonomic Society 9, pp , G. Johansson, Visual motion perception, SciAmer 232, pp , June S. Niyogi and E. Adelson, Analyzing gait with spatiotemporal surfaces, in Computer Vision and Pattern Recognition, J. Little and J. Boyd, Recognizing people by their gait: The shape of motion, Videre 1(2), pp. 1 33, J. Shutler, M. Nixon, and C. Carter, Statistical gait description via temporal moments, in 4th IEEE Southwest Symp. on Image Analysis and Int., pp ,. 6. A. Bobick and A. Johnson, Gait recognition using static, activity-specific parameters, in Computer Vision and Pattern Recognition, pp. I:423 43, R. Tanawongsuwan and A. Bobick, Gait recognition from time-normalized joint-angle trajectories in the walking plane, in Computer Vision and Pattern Recognition, pp. II: , G. Shakhnarovich, L. Lee, and T. Darrell, Integrated face and gait recognition from multiple views, in Computer Vision and Pattern Recognition, pp. I: , J. Hayfron-Acquah, M. Nixon, and J. Carter, Automatic gait recognition by symmetry analysis, in International Conference on Audio- and Video-Based Biometric Person Authentication, pp , 1.. C. BenAbdelkader, R. Cutler, and L. Davis, Motion-based recognition of people in eigengait space, in International Conference on Automatic Face and Gesture Recognition, pp , L. Lee and W. Grimson, Gait analysis for recognition and classification, in International Conference on Automatic Face and Gesture Recognition, pp , A. Kale, A. Rajagopalan, N. Cuntoor, and V. Kruger, Gait-based recognition of humans using continuous HMMs, in International Conference on Automatic Face and Gesture Recognition, pp , R. Collins, R. Gross, and J. Shi, Silhouette-based human identification from body shape and gait, in International Conference on Automatic Face and Gesture Recognition, pp , I. Robledo Vega and S. Sarkar, Representation of the evolution of feature relationship statistics: Human gait-based recognition, IEEE Trans. Pattern Anal. and Mach. Intel. 25, pp , Oct L. Wang, W. Hu, and T. Tan, A new attempt to gait-based human identification, in International Conference on Pattern Recognition, 1, pp , 2.

11 16. P. Jonathon Phillips, S. Sarkar, I. Robledo, P. Grother, and K. Bowyer, Baseline results for the challenge problem of Human ID using gait analysis, in International Conference on Automatic Face and Gesture Recognition, pp , P. Jonathon Phillips, S. Sarkar, I. Robledo, P. Grother, and K. Bowyer, The gait identification challenge problem: Data sets and baseline algorithm, in International Conference on Pattern Recognition, pp , P. Jonathon Phillips, H. Moon, S. Rizvi, and P. Rauss, The FERET evaluation methodology for face-recognition algorithms, IEEE Trans. Pattern Anal. and Mach. Intel. 22(), pp. 9 14,. 19. N. Cuntoor, A. Kale, and R. Chellappa, Combining multiple evidences for gait recognition, in IEEE International Conference on Acoustics, Speech and Signal Processing, 3.. A. Kale, C. B., B. Yegnanarayana, A. N. Rajagopalan, and R. Chellappa, Gait analysis for human identification, in International Conference on Audio- and Video-Based Biometric Person Authentication, A. Sunderesan, A. K. Roy Chowdhury, and R. Chellappa, A hidden markov model based framework for recognition of humans from gait sequences, in IEEE International Conference on Image Processing, D. Tolliver and R. Collins, Gait shape estimation for identification, in International Conference on Audio- and Video-Based Biometric Person Authentication, L. Lee, G. Dalley, and K. Tieu, Learning pedestrian models for silhouette refinement, in International Conference on Computer Vision, L. Wang, T. Tan, H. Ning, and W. Hu, Silhouette analysis-based gait recognition for human identification, IEEE Trans. Pattern Anal. and Mach. Intel. 25, pp , Dec. 3.