(Working Paper, Dec. 2003) Fusion of Hand Geometry and Palmprint Biometrics D.C.M. Wong, C. Poon and H.C. Shen * Department of Computer Science, Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong. {csdavid, carmenp, Helens}@cs.ust.hk Abstract - In this paper, an experimental comparison of the three levels (feature, score and decision) fusion on Hand Geometry and Palmprint Biometrics is reported. Unlike other multimodal biometric systems, only one contact-free device is required to capture a single hand image. Both personal identification and authentication experiments were carried out for comparison. Our experimental results on the image database from 170 users (3400 images) agreed on the belief that multimodal biometrics outperform individual single biometrics. The results also demonstrated that fusion at the score level is an upper bound of both feature and decision level in terms of recognition accuracy. Keywords: Biometrics; multimodal; Hand Geometry, Palmprints, Personal Identification; Personal Identity Verification. 1 Introduction A single-mode biometric system is a biometric system that utilizes only one biometric feature. Examples of single mode biometrics are fingerprints, face, iris and palmprint [1][2][3]. Each of these techniques relies on a certain unique physiological feature of an individual. On the other hand, a multimodal biometric system is a biometric system that utilizes features from different biometric modalities and takes the advantages of each of these biometrics [4][5][6]. It is believed that multimodal biometric systems outperform * Corresponding Author: helens@cs.ust.hk - 1 -
single-mode biometric systems. The accuracy of a system is improved by combining different biometrics. Oftentimes, when one biometrics fails to recognize a person, the other biometrics can recognize the person successfully. In this paper, we shall demonstrate the effect of fusing hand geometry and palmprint biometrics. The advantage of this combination is that features of both hand geometry and palmprint can be extracted from a single hand image. In case the palmprints is affected by a cut on the palm, the individual s hand geometry will still serve to identify the individual. Unlike other multimodal biometric systems (e.g. face and voice), only one non-contact device is used in this multimodal system. This reduces the cost of system implementation and speed up the process of image acquisition for a multimodal biometrics. 1.1 Related research Hand geometry has been shown to be the first biometrics that are practically in use across a variety of real world applications [7]. The first automate hand geometry recognition system was introduced in the middle of 19 th century [8]. Related work has been done by Sanchez- Reillo [9] and Ross [10]. They used a CCD camera to capture the image of the back of the hand. Five to six pegs were used to fix the hand position in order to reduce the variation in the images captured. Their feature vector comprises of 20 components. Authentication and identification experiments have been performed on the hand image dataset. Sanchez- Reillo s experimental results [9] show up to 97 percent success in classification among 20 users, using 10 images per user, 5 for training and another 5 for testing. Ross s experiment [10] show an optimum verification result in which the Equal Error Rate (EER) is around 4.8% of 360 images taken from 50 users. - 2 -
Palmprint is the biometrics that is most similar to fingerprint biometrics. Like fingerprint biometrics, there are ridges and valleys on the surface of the palm. The commonly used features of palmprint are the principal lines and wrinkles [11]. Image enhancement is required to remove noise [12] in the input image or to make it smooth. Lines of palmprint can be obtained by applying line detection, wavelet or Gabor filter. Most of them apply wavelet [13] or Gabor [14] filter to the image, and divide the image with block size N x N, then encode the feature vector with these information. A related work on the fusion of hand geometry and palmprint biometrics was done by Kumar et al.[16]. Hand geometry and palmprint features are extracted from the same hand image. They found that fusion at the decision level of these two biometrics gives better results. 1.2 Outline of the paper In this paper, we compare different fusion schemes by experiments. We consider the three levels of fusion, namely, the feature level, the score level and the decision level. Both personal identification and authentication experiments were carried out for evaluation. Details of the three fusion schemes are described in Section 2. In Section 3, we describe the dataset and matching modules used in our experiments. Experimental results are then reported. A conclusion is drawn in Section 4. 2 Fusion schemes In our proposed system, hand geometry and palmprint are captured as a single image. We treat the left and right hand of the same person as two different biometrics, thus resulting in four biometrics for the fusion: left hand geometry, right hand geometry, left palmprint and right palmprint. In this paper, we are going to evaluate the effect of fusion at different - 3 -
levels (feature, score and decision as shown in Figure 1) and with different combinations of the four biometrics. The idea of fusion at the feature level is to concatenate the feature vectors of different biometrics. Fusion at the score level requires the calculated matching scores from different biometrics to make a final decision. In the decision level fusion scheme, matching score and decision are calculated independently for different biometrics. The final decision is made by a majority vote within the different decisions made by different biometrics. Details of these three schemes are described in the following paragraphs. 2.1 Feature level fusion Feature vectors from different biometrics are concatenated to form a larger feature vector for recognition. The training and testing processes are based on these concatenated feature vectors. In our research, we use hand geometry and palmprint features to perform the pattern recognition. A brief description of the feature extraction process for hand geometry and palmprint is presented in the following paragraphs. Image preprocessing is required before the extraction of palmprint or hand geometry features. The first step in image preprocessing is to convert the grey level image to a binary one. This is necessary for the image alignment algorithm and feature extraction. We cannot apply a fixed threshold value to every image because of different lighting conditions or skin colors. Thus threshold value will be identified from the histogram of each image. Based on the threshold value, the corresponding binary image will be obtained. Our image acquisition device is contact-free without pegs to align the fingers and hand positions. Therefore, image alignment is essential before features can be extracted. We have developed an algorithm to compute the angle of rotation to align all hand images along the same direction. Figure 2(a) is the schematic diagram of the alignment algorithm. The angle - 4 -
of rotation θhas to be computed before the rotation process. We scan the binary image from top to bottom to locate the two gap points as shown in Figure 2(a). The angle of 1 Y2 Y1 rotation is calculated by the equation: tan ( ). Then, the whole image can be X X rotated by this calculated angle θ (Figure 2(b)). 2 1 2.1.1 Feature extraction of hand geometry Extraction of hand geometry features is based on the binarized hand image. Some control points are obtained before the features are being measured (Figure 3(a)). For example, finger length can be obtained by calculating the distance between the finger-tip and the finger-end point. Palm length is considered as the distance between the finger-end point of the middle finger and the wrist point. After detecting all these essential points, we use them to calculate the seventeen feature values that will be used for personal identification/authentication. Figure 3(b) gives the length features (in terms of number of pixels) extracted from the binary hand image. The feature vector of the hand geometry comprises of four finger lengths (FL), eight finger widths (FW), hand length (HL), palm width (PW), palm length (PL), the contour length of the hand (HCL) and the area of the hand (HA). 2.1.2 Feature extraction of palmprint The palm area (region of interest) has to be located before further feature extraction. With referencing to the locations of the gaps between fingers, the palm is duly rotated and the maximum palm area is then located. The gap between the middle and ring fingers is used to determine the selected region of the palm so as to minimize translation errors in different palm images of the same person (Figure 4). - 5 -
A 2D Gabor filter is then applied to the region of interest in the grey level image to enhance the principal lines. A 135 o Gabor filter with frequency equal to 1/(5*x) is used. The variable x is a major parameter used in the adaptive filtering process, and the default value is 1.5. If the palm lines are too light to be identified by the default value, our algorithm will automatically adjust the value of x to produce clear palm line features. A degree of 135 o is chosen because we found that almost all principal lines of the left hand are in this orientation. Therefore, a 45 o Gabor filter is suitable for right hand image. An enhanced palm image after applying the Gabor filter is shown in Figure 5(b). The principle lines are clearer, the background and shadow area are suppressed. A Sobel edge filter is applied to this enhance palm image to extract the principal lines and wrinkles. The resulting image is a binary image contains only the principal lines and wrinkles (Figure 5(c)). With the extracted lines image, the image is divided into 11x11 overlapping blocks. For each block, the number of line pixels is considered as a feature. The feature vector of the palm image comprises of the width, the height and the total line pixels of the palm image followed by the 121 sum of line pixels within each block. Therefore, there are totally 124 features in one palmprint feature vector. 2.1.3 Fusion of feature vectors Feature vectors combination is the main idea of fusion at the feature level. Feature vector extracted from different biometrics are being concatenated to form a new feature vector. The compatibility of the feature vector from different biometrics is an important factor to be considered. In our case, the feature values from the hand geometry and palmprint are both in terms of the number of image pixels. Thus, fusion at the feature level is by concatenating the two feature vectors to form one feature vector with 141 features. - 6 -
2.2 Score level fusion Score level fusion is achieved by combining the matching scores calculated from different biometrics. Each individual biometrics has its feature vector and matching module. Matching score is calculated individually with the corresponding feature vector and matching module. There are several ways, such as the sum and weighted sum rules, to combine these scores. In this paper, we concentrate only on the weighted sum method. The reason is that the trivial sum rule does not take the performance of each individual biometrics into consideration. For example, if one of the biometrics (e.g. Iris recognition) has relatively higher accuracy and another biometrics (e.g. face recognition) has fairly low accuracy, the sum of the two normalized scores may result in poorer performance. An appropriate weighting among them is essential in order to have a balance between different biometrics and achieve higher accuracy. 2.3 Decision level fusion At the decision level fusion scheme, each individual biometric has its own feature vector and matching module. Feature extraction and matching score calculation are carried out individually. The decisions from different biometrics are employed for fusion. The most commonly used fusion rule is by majority voting. The idea of majority vote is that we keep counting the decisions made by different biometrics and set the majority decision as the final decision. 3 Experiments and results In our experiments, we evaluated on the 3400 samples database, which is collected from 170 persons. For each person, 10 left and 10 right hand images are captured with slightly - 7 -
variation of the post (not more than 20 o rotation of hand). Samples of the hand images of one person are shown in Figure 6. In our system, both hand geometry and palmprint features are extracted from a single image. Since the feature extraction of hand geometry and palmprint are independent, these two processes can be run in parallel. The total time required for both feature extractions are about 2 seconds (without software and hardware optimization). We also consider left and right hand as two different biometrics. Thus, there are four biometric modals for fusion in our experiments. The performance of individual biometrics is shown for comparison. We perform five different multimodal combinations: A) left and right hand geometry, B) left and right palmprint, C) left hand of hand geometry and palmprint, D) right hand of hand geometry and palmprint and E) all of the four biometrics. Fusion is also performed at all three levels. Our experiments are divided into two major parts: personal identification and authentication. Personal identification refers to the problem of finding out the identity of a subject (Who am I?). Personal authentication refers to the problem of confirming a person s claimed identity (Am I who I claim I am?). At the feature level, the feature vector of the multimodal biometrics is a concatenation of the individual feature vectors. At the score level, we employed the weighted sum rule. Weights are calculated from the training set data and used to recognize test samples. Therefore, the size of the training set should not be too small, otherwise, it may give an unfair weighting. In our experiment, we use the training set, which contains 1700 samples from 170 classes for the weight calculation. In our case, we actually have two biometrics: hand geometry and palmprint. The matching scores calculated from the two biometrics are first normalized to the range 0 to 10. Then, the optimal ratio of the two weights (R = w 1 /w 2 ) - 8 -
is measured by achieving the maximum classification rate of 2-fold cross-evaluation with score fusion on the training set by varying the ratio R from 0 to 1 with step size 0.01: max 100 0 i i = ( Accuracy _ of _ 2 fold R = ) 100 This calculated weights are used in the experiments of personal authentication and identification. At the decision level, the majority vote rule is used. For the personal identification experiment, there are three to five decisions come out from each biometric. All calculated matching scores are sorted in each biometrics. The top three or five are output as the set of decision. Different sets of decision are collected from different biometrics, and the majority vote is carried out with all these collected decisions: Assign the final decision to C i if it is the majority vote n max i= 1 ( Set _ of _ decision Ci ) where n is the number of classes, C i is a specify class and Set_of_decision is all the collected decisions. The weights calculated in score level fusion can be used to break ties when equal votes. For the personal authentication experiment, only one decision (True/False) comes from each biometric. Majority vote is carried out with all these collected decisions as in personal identification experiment. 3.1 Personal identification The identification experiments were conducted in two ways: 1) 10-fold cross-validation, 2) 2-fold cross-validation. In general, k-fold cross-validation is a model evaluation method [15]. The dataset is split into k mutually exclusive subsets of approximately equal size. One of the subsets is chosen to be the test data and the other (k-1) subsets are used for training. The classification process will be performed k times with different testing subset each time. - 9 -
The total classification rate will be reported. The matching module used in this experiment is the knn classifier with Euclidian distance method. The cross-validation results of all the fusion choices and schemes are shown in Table 1. The first four rows show the performance of individual biometrics. All the best results from each row are shown in the first column (10-fold with k=1). The results of 10-fold are better than 2-fold is because more training samples are used in 10-fold than in 2-fold. knn (k=1) produce the best results because our training set size is fairly small (e.g. 5-9 samples per class). According to the experimental results, multimodal biometrics performs better than the individual biometrics. This shows that multimodal biometrics have the ability to improve the recognition accuracy over individual biometrics. This result is shown clearly at the score level fusion of all the four biometrics. We have a 100% classification rate with 2-fold and 10-fold cross-validation over 170 persons, which is impossible to be achieved by any of the individual biometrics. In general, fusion at the feature level produces better results than the individual biometrics. For example, fusion of the left and right hand at the feature level give 99.5% for 10-fold with k equal to 1, which is much higher than the individual results 96.1% and 98.2%. Fusion at the score level gives better performance than individual modality. In particular, for the fusion of left and right hand geometry in 2-fold experiment, the improvement is from 87.9% and 91.5% to 97.5%. The results of multimodal biometrics are better than single biometrics at the decision level especially when all the four biometrics are combined. The improvement is from 87.9%, 91.5%, 96.4% and 96% to 99.9% in the 2-fold experiment. Note that the value k has a different meaning for fusion at the decision level. It refers to the number of output decisions made by each biometric. The results show that higher - 10 -
classification accuracy can be achieved with k equal to 3 in general. It is reasonable that fewer choices for voting will give poor results and the outlier s vote may affect the results. A more detail comparison among the three level fusion schemes is shown in Table 2, which contains the best classification rate that can be achieved by each fusion scheme in the 2-fold experiment. By comparing the results of fusion at the feature level and decision level, it shows that the classification rate for fusion at the decision level is slightly higher than that of fusion at the feature level. For example, result of decision level fusion has a 0.4% improvement over feature level fusion for multimodal E. For fusion at the score level, it gives the highest classification rate and outperform feature and decision levels fusion. In particular, for the multimodal E, result of score level fusion is 100%, which is the best result that can be achieved. This observation is acceptable and reasonable. It is because fusion at the score level is more flexible than at the feature and decision level. Unlike feature level fusion, different biometrics can have its own best-fit matching module, and weights are introduced to balance the difference in performance of different biometrics. Moreover, we believe that the scores are more valuable and representative than just a vote as at the decision level. 3.2 Personal Authentication Authentication is the step to confirm a claimed identity of the user. Features are extracted from the input image and being compared to the records of the claimed identity in the database. The matching score for each record will be calculated and used for making judgment. The aim of this experiment is to evaluate the effect on performance with multimodal biometrics. The 3400 samples dataset are split into training set and testing set with equal size (5 samples per class). For each sample in the testing set, authentication is performed to each - 11 -
class in the corresponding training set. The sample is accepted if the matching score is lower than the current threshold, and it is rejected if the matching score is higher than the current threshold. Therefore, 850 genuine users and 143650 imposters are used to evaluate the Equal Error Rate (EER). From the evaluation results (Table 3), we can see that the EER of multimodal biometrics are lower than the individual biometrics on average. The overall accuracy is improved with fusion of different biometrics. According to the results of feature level and decision level fusion, the EER of feature level fusion is lower than that of decision level fusion in general, which is different from the observation in personal identification experiment. It is because fusion at the decision level in personal authentication is different. The decision made from different biometrics is only a single True/False answer. Therefore, there are fewer decisions for voting in personal authentication and the accuracy is dropped. An increment of accuracy for decision level fusion is shown in multimodal E, which is because more decisions are available for voting with more biometric modalities. Similar to the observation in personal identification, fusion at the score level outperforms the other two schemes, especially demonstrated in multimodal C. The EER of score level fusion is 1.5%, while the EER of feature and decision level fusion are 2.1% and 2.3%, respectively. 4 Conclusions The objective of this work is to compare the performance of the three fusion schemes on palmprint and hand geometry biometrics, and to achieve higher accuracy that may not be possible with single biometric alone. The three fusion schemes are feature level, score level and decision level fusion. Personal identification and authentication experiments were carried out with a 170 persons (3400 samples) database to evaluate the effect of fusion. Both identification and authentication results demonstrate that multimodal biometrics - 12 -
outperform every individual biometrics. Our results show that the score level fusion scheme, with weighted sum rule, achieves better performance than those for fusion at the feature and decision level. The best-achieved results are 100% classification rate for both 10-fold and 2- fold cross-validation, and 0.7% EER. To achieve higher accuracy in the future, we shall extend our work by including more biometric modalities such as face and voice biometrics. Acknowledgments The research work is supported by the Sino Software Research Institute (SSRI) grant from HKUST, grant number SSRI01/02.EG12. References [1] A. K. Jain, R. Bolle and S. Pankanti, Biometrics: Personal Identification in Networked Society, Kluwer Academic, USA, 1999. [2] X. Tan, B. Bhan, On the fundamental performance for fingerprint matching, IEEE Proceedings. Computer Vision and Pattern Recognition, vol. 2, pp. 499-504, June 2003. [3] W. K. Kong, D. Zhang, Palmprint texture analysis based on low-resolution images for personal authentication, International Conference on Pattern Recognition, vol. 3, pp. 807-810, Aug, 2002. [4] S. K. Dahel, Q. Xiao, Accuracy performance analysis of multimodal biometrics, IEEE System, Man and Cybermetics Society, pp 170-173, June 2003. [5] A. Ross, A. Jain and J-Z. Qian, "Information Fusion in Biometrics", 3 rd Audio- and Video-Based Biometric Person Authentication, pp. 354-359, 2001. [6] S. Ben-Yacoub, Y. Abdeljaoued and E. Mayoraz, Fusion of face and speech data for person identity verification, IEEE Transaction on Neural Network, vol. 10, pp. 1065-1074, Sep 1999. [7] J. Ashbourn, Biometrics: Advanced Identity Verification: the complete guide, Springer-Verlag London Limited, Britain, 2000. [8] A. K. Jain, R. Bolle and S. Pankanti, Biometrics: Personal Identification in Networked Society, Kluwer Academic, USA, 1999. [9] R. Sanchez-Reillo, Biometric Identification through Hand Geometry Measurements, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1168-1171, 2000. [10] A. K. Jain, A. Ross, and S. Pankanti, A prototype hand geometry-based verification system, in Second International Conference on Audio and Video-based Biometric Person Authentication, (Washington, D.C., U.S.A.), pp. 166-171, Mar 1999. [11] S. Wei and D. Zhang, Palmprint verification: an implementation of biometric technology, International Conference on Pattern Recognition, vol.1, pp. 219-221, - 13 -
1998. [12] J. Funada, N. Ohta, M. Mizoguchi, T. Temma, K. Nakanishi, A. Murai, T. Sugiuchi, T. Wakabayashi and Y. Yamada, Feature extraction method for palmprint considering elimination of creases, International Conference on Pattern Recognition, vol. 2, pp. 1849-1854, 1998. [13] X. Q. Wu, K. Q. Wang and D. Zhang, Wavelet based palmprint recognition, International Conference on Machine Learning and Cybernetics, vol. 3, pp. 1253-1257, 2002. [14] D. Zhang et al., Online Palmprint Identification, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1041-1050, 2003. [15] R. Kohavi, A study of cross-validation and bootstrap for accuracy estimation and model selection, in Proceedings of the International Joint Conference on Arti cial Intelligence, pp. 1137-1143, 1995. [16] A. Kumar, D.C.M. Wong, H.C. Shen, A.K. Jain, "Personal Verification using Palmprint and Hand Geometry Biometrics," Proceedings of the fourth International Conference on audio- and video-based biometric personal authentication, June 2003. - 14 -
Figures Figure 1: Three fusion schemes (a) (b) Figure 2: Hand image rotation process: (a) Original image, two gap points are detected and the rotation angle is calculated (b) Rotated hand image - 15 -
(a) (b) Figure 3: Extraction of hand features: (a) Control points detected in the hand image (b) Length features calculated with control points Four Fingers Located Gap- Between- Fingers Extracted Palm Region Figure 4: Region of interest (Palm region) - 16 -
(a) (b) (c) Figure 5: An example of Gabor filtering and line detection: (a) Original palm image, (b) Image obtained after applying Gabor filter, (c) Detected palm lines image Figure 6: Samples of the captured hand images from the same person - 17 -
Tables A B C D E Table 1: Correct Classification Rate for the 170 people database (3400 samples) Modality knn (k=1) 10-fold knn (k=3) knn (k=5) knn (k=1) 2-fold knn (k=3) knn (k=5) Hand Geometry (L) 96.1 95.8 93.8 89 87.9 85.8 Hand Geometry (R) 98.2 97.2 96 91.9 91.5 89.2 PalmPrint (L) 99.3 99 98.7 96.8 96.4 94.7 PalmPrint (R) 99.1 98.8 98.4 97.5 96 94.4 Hand Geometry (L+R)-feature lv 99.5 99.4 99 97.5 97.5 96.5 Hand Geometry (L+R)-score lv 99.5 99.4 99 97.5 97.5 96.5 Hand Geometry (L+R)-decision lv 98.2 99.2 99.4 91.9 96.5 96.7 PalmPrint (L+R)-feature lv 99.8 99.7 99.7 99.3 99.4 99 PalmPrint (L+R)-score lv 99.8 99.7 99.7 99.3 99.4 99 PalmPrint (L+R)-decision lv 99.3 99.8 99.8 96.8 99.2 99.4 HG(L)+Palm(L)-feature lv 99.3 99.1 98.8 97 96.4 94.7 HG(L)+Palm(L)-score lv 99.6 99.6 99.5 98.2 98.2 97.5 HG(L)+Palm(L)-decision lv 99.3 99.5 99.4 96.8 97.9 97.9 HG(R)+Palm(R)-feature lv 99.2 98.8 98.7 97.5 96 94.5 HG(R)+Palm(R)-score lv 99.5 99.4 99.2 98.4 97.7 96.7 HG(R)+Palm(R)-decision lv 99.1 99.5 99.4 97.5 98.1 97.8 Fusion All in feature lv 99.8 99.8 99.8 99.5 99.5 99.1 Fusion All in score lv 100 100 99.9 100 100 99.8 Fusion All in decision lv 99.9 100 99.9 99.5 99.9 99.7 Table 2: Best results of 2-fold cross-validation for each fusion scheme Feature level Score level Decision level A 97.5 97.5 96.7 B 99.4 99.4 99.4 C 97.0 98.2 97.9 D 97.5 98.4 98.1 E 99.5 100 99.9-18 -
A B C D E Table 3: EER for our 3400 samples image database EER (Evaluation) Hand Geometry (L) 3.6% Hand Geometry (R) 3.3% PalmPrint (L) 2.2% PalmPrint (R) 1.9% Hand Geometry (L+R)-feature lv 2.5% Hand Geometry (L+R)-score lv 2.5% Hand Geometry (L+R)-decision lv 5.5% PalmPrint (L+R)-feature lv 1.1% PalmPrint (L+R)-score lv 1.1% PalmPrint (L+R)-decision lv 1.2% HG(L)+Palm(L)-feature lv 2.1% HG(L)+Palm(L)-score lv 1.5% HG(L)+Palm(L)-decision lv 2.3% HG(R)+Palm(R)-feature lv 1.8% HG(R)+Palm(R)-score lv 1.6% HG(R)+Palm(R)-decision lv 2.0% Fusion All in feature lv 1.0% Fusion All in score lv 0.7% Fusion All in decision lv 0.8% - 19 -