Biometrics Technology: Multi-modal (Part 2) References: At the Level: [M7] U. Dieckmann, P. Plankensteiner and T. Wagner, "SESAM: A biometric person identification system using sensor fusion ", Pattern Recognition Letters, Volume 18, Issue 9, pp. 827-833, 1997. At the Feature Level: [M8] Kyong Chang, K.W. Bowyer, S.Sarkar, B. Victor, Comparison and Combination of Ear and Face Images in Appearance-based Biometrics, IEEE Trans. on PAMI, Vol. 25, No. 9, Sept. 2003, pp.1160-1165. At the Level: [M9] Brunelli, R., Falavigna, D., "Person identification using multiple cues ", IEEE Trans. on PAMI, Vol. 17, No. 10, pp. 955-966, 1995. [M10] L. Hong and A.K. Jain, "Integrating Faces and Fingerprints for Personal Identification", IEEE Trans. on PAMI, Vol. 20, No. 12, pp. 1295-1307, 1998. [M11] Mahesh Viswanathan, Homayoon S.M. Beigi and Fereydoun Maali, "Information Access Using Speech, Speaker and Face Recognition ", ICME2000, New York City, New York, USA, July 2000. [M12] Chang, K.I.; Bowyer, K.W.; Flynn, P.J., Multimodal 2D and 3D biometrics for face recognition Analysis and Modeling of Faces and Gestures, 2003. AMFG 2003. IEEE International Workshop on17 Oct. 2003 Pages:187 194. [M13] Y. Wang, T. Tan and A. K. Jain, "Combining Face and Iris Biometrics for Identity Verification", Proc. of 4th Int'l Conf. on Audio- and Video-Based Biometric Person Authentication (AVBPA), pp. 805-813, Guildford, UK, June 9-11, 2003. [M14] A. Jain, K. Nandakumar and A. Ross, Normalization in Multimodal Biometric Systems, Pattern Recognition, 2005. w03-multi-modal (P2) Biometrics - Summer 2006 1
Three possible levels of Integration of Multiple Biometrics User Database Biometric Feature Vector Biometric Feature Vector Database Final Database Final Representation Final w03-multi-modal (P2) Biometrics - Summer 2006 2
Fusion at the Level At this level, each biometric system will arrive at a decision. The final decision is made based on some techniques, e.g. by majority voting. Understandably, this type of fusion is considered less accurate and rigid. User Database Biometric Feature Vector Biometric Feature Vector Database Final w03-multi-modal (P2) Biometrics - Summer 2006 3
Fusion at the Level [M7] Three different biometric cues from two different data sources: Face from CCD camera (static) and video recording (Fig. 3) from the still image, and by locating the positions of the eye and mouth, extraction of the face image from the video sequence can be done feature vector of 31329 floating point values Lip movement from 1-min. video recording (dynamic) (Fig. 4) A sub-region of the face image is extracted from the video sequence (128x128 block). The optical flow is calculated between two consecutive frames and stored in 16 vector fields of 32x32 blocks feature vector of 16384 floating point values. Voice from microphone recording about 1000ms (dynamic) (Fig. 5) using 32 windows and 1024 samples in each window. Dimension of the power spectrum is reduced feature vector of 4096 floating point values. w03-multi-modal (P2) Biometrics - Summer 2006 4
Fusion at the Level [M7] Integration: Classifier - Synergetic Computer (existing) based on an algorithm called MELT Euclidean distance and 1-NN is the classification rule is 2-from-3. Results: Tables 1 and 3 w03-multi-modal (P2) Biometrics - Summer 2006 5
Fusion at the Feature Level Assuming that the features extracted from single biometrics are independent from each other, they are combined to form a new feature representation (vector). Input Type 1 Biometric Feature Type N Biometric Feature Database Representation Final w03-multi-modal (P2) Biometrics - Summer 2006 6
Fusion at the Feature Level Ref.[M8] [M8] Comparison and Combination of Ear and Face Images in Appearance-based Biometrics Based on eigen-faces and eigen-ears and Principal Component Analysis (PCA), Fusion: Face images and Ear images are normalized by PCA to 130x150 Histogram equalization is also performed. Eigenvalues and eigenvectors are computed eigenfaces and eigen-ears representing the feature Results: Fig. 3, 4, 5 w03-multi-modal (P2) Biometrics - Summer 2006 7
Fusion at the Level Ref.[M9 M13] Each system provides the matching score. User Database Biometric Feature Vector Final Biometric Feature Vector Database w03-multi-modal (P2) Biometrics - Summer 2006 8
Fusion at the Level Ref.[M9] [M9] Person Identification Using Multiple Cues Based on acoustic and visual (face) features, 5 biometrics, i.e. static, dynamic acoustic features and eyes, nose and mouth features, produce nonhomogeneous lists of scores. Two approaches: Measurement level scores are normalized to the interval of (0,1) by the tanh function (Eqtn. 15), (Figure 5) Integration is using a geometric average (Eqtn. 16, 17) an integrated score. For verification: A score vector (18 dimensions) is generated as input to the linear classifier (Fig. 6) is made according to Eqtn. 23, 24. Result: Table I (identification); Table III w03-multi-modal (P2) Biometrics - Summer 2006 9
Fusion at the Level Ref.[M9] [M9] Person Identification Using Multiple Cues Two approaches: (cont d) Hybrid level the mapping from the scores of the classifiers and their ranks into the interval of (0,1) is approximated by using the HyperBF network. Final decision based on cross-validation using threshold value. Result: Figures 9, 10 w03-multi-modal (P2) Biometrics - Summer 2006 10
Fusion at the Level Ref.[M10] [M10] Integrating Faces and Fingerprints for Personal Identification Based on Eigen-faces and Minutiae Finger Print features Fusion: Assume the score from each system is a set of possible labels associated with some confidence measures. Confidence measure can be characterized by FAR which is indicated by the impostor score distribution. Estimate of the Finger print Imposter dist. -Eqtn. (9) Estimate of the face imposter dist. Eqtn. (10) Composite imposter dist. Eqtn. (20) Fusion Criterion Eqtn. (21) Results: Fig. 10; Table 1; Fig. 11. w03-multi-modal (P2) Biometrics - Summer 2006 11
Fusion at the Level Ref.[M11] [M11] Information Access using Speech, Speaker and Face Recognition Based on Speaker model (mixture of Guassian distributions) and face (29 landmarks) Fusion: weighted sum of scores Compute the linear combinations of the audio and face class assignments, Fig. 3. The weights are based on the slopes of the linear lines. Results: Fig. 4. w03-multi-modal (P2) Biometrics - Summer 2006 12
Fusion at the Level Ref.[M12] [M12] Multi-Modal 2D and 3D Biometrics for Face Recognition Based on 2D and 3D eigen-faces and Principal Component Analysis (PCA), Fusion: A weight is estimated based on the distribution of the top three ranks in each space (i.e. the 2D and the 3D). The decision is made as follows: Distance scores are computed within each of the 2D and 3D training set. A confidence measure is computed using the three distances in top ranks, i.e. (2 nd dist. 1 st dist)/(3 rd dist. 1 st dist.) The confidence measure is used as weights in the sum rule. Results: Figures 6, Table 2. w03-multi-modal (P2) Biometrics - Summer 2006 13
Fusion at the Level Ref.[M13] [M13] Combining Face and Iris Biometrics for Identity Verification Based on Eigen-faces and Iris Fusion: (3 approaches) Weighted Sum FAR and FRR are the bases in calculating the weights, (Eqtn. 2, 3) Fisher Discriminant Analysis s form face and Iris are treated as feature vector in the 2D space, a linear separation boundary can be found to separate the genuine and impostor classes. Radial-based Function NN again scores from the face and iris system is treated as a 2D vector. RBF NN is used for classification. Results: Fig.3 distributions; Table 1; Fig. 4 w03-multi-modal (P2) Biometrics - Summer 2006 14
Normalization in Multimodal Buiometric Systems Ref.[M14] scores produced by various single modality biometric systems are usually heterogeneous, score normalization is needed to transform them into one common domain before combining them for decision making. Study on various normalization techniques and fusion rules. Based on face, fingerprint and hand-geometry biometrics. w03-multi-modal (P2) Biometrics - Summer 2006 15
Summary Three levels in biometric fusion, namely feature, score and decision level. At each level, fusion can be performed on the same biometrics and/or on different biometrics. At the current research findings, fusion at the score level performs best. w03-multi-modal (P2) Biometrics - Summer 2006 16