Proceedngs of Student/Faculty Research Day, CSIS, Pace Unversty, May 6th, 2005 On Evaluatng Open Bometrc Identfcaton Systems Mchael Gbbons, Sungsoo Yoon, Sung-Hyuk Cha and Charles Tappert mkegbb@us.bm.com, {scha, ctappert}@pace.edu Abstract Ths paper concerns the generalzablty of bometrc dentfcaton n open systems. Many researchers have clamed hgh dentfcaton accuraces on closed system consstng of a few hundred or thousand members. Here, we consder what happens to these closed dentfcaton systems as they are opened to non-members. We clam that these systems do not generalze well as the non-member populaton ncreases. To support ths clam, we frst take a look at the vsualzaton of pattern classfcaton usng Support Vector Machnes (SVM), Nearest Neghbor (NN) and Artfcal Neural Network (ANN). Next, we present expermental results on wrter and rs bometrc databases usng the afore mentoned classfers. We fnd that system securty (1-FAR) decreases rapdly for closed systems when they are tested n open-system mode as the number of non members tested ncreases. We also fnd that, although systems can be traned for greater closed-system securty usng SVM rather than NN classfers, the NN classfers are better for generalzng to open systems due to ther superor capablty of rejectng non members. Keywords Bometrc dentfcaton, Wrter dentfcaton, Irs dentfcaton, Nearest Neghbor, Support Vector Machnes 1 INTRODUCTION Bometrc applcatons are becomng more common and acceptable n today's socety. Technology contnues to mprove, provdng faster processors, smaller sensors and cheaper materals, all of whch are contrbutng to relable, affordable bometrc applcatons. The most common use of bometrcs s for verfcaton. In bometrc verfcaton systems, a user s dentfed by an ID or smart card and s verfed by ther bometrc,.e., a person s bologcal or behavoral characterstc such as ther fngerprnt, voce, rs, or sgnature. Ths s analogous to a user at an ATM machne usng a bank card to dentfy and a PIN to verfy. Another use of bometrcs s for dentfcaton, whch s the focus of ths paper. Identfcaton can be appled n a closed system such as employee postve dentfcaton for buldng access, or n an open system such as a natonal ID system or negatve dentfcaton for passenger screenng at an arport. Postve bometrc dentfcaton, a 1-to-many problem, s more challengng than verfcaton, a 1-to-1 problem. As stated n [1], "postve dentfcaton s perhaps the most ambtous use of bometrcs technology." There have been many promsng results reported for closed dentfcaton systems. Although hgh accuraces have been reported n wrter, rs, and hand geometry studes [4, 10, 12, 15, 17], these accuraces may lead to a false mpresson of securty. One may ask whether there really are any stuatons that correspond to closed worlds[1]. For example, take an employee dentfcaton system. Can t be guaranteed that a bometrc of a guest (a non-member) vstng the faclty does not match one of an employee (a member)? Ths paper wll nvestgate the generalzablty of bometrc dentfcaton as t pertans to the securty of an open system. Our hypothess s that the accuraces reported for closed systems are relevant only to those systems and do not generalze well to larger, open systems contanng non-members. Snce t s mpractcal to test a true populaton, we use a reverse approach to support the hypothess. We wll work wth a database M of m members, but assume a closed system of m ' members, where m' < m, and tran the system on the m ' members. We then have m m' members to test how well the system holds up when non-members attempt to enter the system. Ths approach s used on two bometrc databases, one consstng of wrter data and the other of rs data. In secton 2 of ths paper, postve dentfcaton and the assocated error rates wll be explaned. In secton 3, the bometrc databases and the pattern classfcaton technques used n ths paper wll be descrbed. In secton 4, the statstcal experments to support the hypothess are descrbed and observatons presented. Secton 5 concludes wth a summary and consderatons for future work. 2 ERROR RATE TYPES IN BIOMETRIC IDENTIFI- CATION Consder the postve dentfcaton model. Postve dentfcaton refers to determnng that a gven ndvdual s n a member database[1]. Ths s an m-class classfcaton problem that s, gven data from m subjects n a bometrc database M = { s1, s2, L, s m }, the problem s to dentfy an unknown bometrc sample d from a person,, where s q D5.1
s q M. In ths model, a classfer can be traned based on d all exemplars n M to fnd decson boundares, e.g., support vector machne. If a smlarty-based classfer such as a nearest neghbor s used, an unknown sample d s compared to each of M. The error rate n ths model s smply a number of msclassfed nstances dvded by the testng set sze. We clam that a classfer wth the lowest error rate s not necessarly the best for securty, and that classfer desgners mght consder the followng three types of error rates. In ths paper, we wll pay close attenton to the Securty measurement as we test our hypothess. Consder an unknown bometrc sample d from a person, sq, where sq M. If ths bometrc data of a non member enters drectly to the above model, can t be classfed correctly? If the classfer has no reject capablty, the unknown wll be classfed nto one of the decson regons or as the closest matchng subject. However, f the classfer has a reject capablty, the number of classes n M becomes m +1,.e., m member classes + 1 reject class. Therefore, f the questoned nstance s n a reject area n SVM or the closest match s outsde the nearest neghbor thresholds, the unknown wll be classfed as none of the members. Ths study nvestgates the reject capablty of two classfers: support vector machne and nearest neghbor. In the later scenaro wth members and non-members, there are three knds of error. A false reject, FR, error occurs when a classfer dentfes an unknown bometrc sample d from a person, sq, where sq M, as a reject. The other errors are false accepts, FA, of whch there are two types those that can occur between members of the system, FA (1), and those that can occur as non-members enter the system, FA (2). FA(1) occurs when a classfer dentfes an unknown bometrc sample d from a person, sq to s where sq, s M and sq s. FA(2) occurs when a classfer dentfes an unknown bometrc sample d from a person, sq to s, where sq M and s M. Fgure 1 llustrates these three error types. The frequences at whch the false accepts and false rejects occur are known as the False Accept Rate (FAR) and the False Reject Rate (FRR), respectvely. These two error rates are used to determne the two key performance measurements of a bometrc system: convenence and securty [1]: Convenence = 1 FRR Securty = 1 FAR (1) Fgure 1. (a) The graph dsplays classfcaton boundares for a hypothetcal two-member database usng the Nearest Neghbor classfer wth threshold t. False accepts and false rejects can occur between members of the system, and false accepts can also occur as non-members enter the system. (b) Same as (a) but wth a three-member database and SVM classfer. 3 BIOMETRIC DATABASES AND CLASSIFIERS Two bometrc databases are used to support our clams n ths study: the wrter and rs bometrc databases. Although there are many classfers to choose from n the feld of pattern classfcaton, we used two pattern classfcaton technques: Support Vector Machnes (SVM) and Nearest Neghbor. 3.1 Databases In a prevous study, Cha et al. [3] studed the ndvdualty of handwrtng usng a database of handwrtng samples D5.2
from 841 subjects representatve of the Unted States populaton. Each subject coped three tmes a source document contanng 156 words carefully constructed to have each letter of the alphabet used n the startng (both upper and lowercase), mddle (lowercase), and endng poston (lowercase) of a word. Each document was dgtzed and features were extracted at the document, word, and character level. For the purposes of ths study, we used the same database but focus only on the document features: entropy, threshold, number of black pxels, number of exteror contours, number of nteror contours, slant, heght, horzontal slope, vertcal slope, negatve slope, and postve slope. A detaled explanaton of these features can be found n [4]. From the rs bometrc mage database [9], we selected 10 left bare eye samples of 52 subjects. In comparson to the wrter database, the rs database has many fewer subjects, but a much larger number of samples per subject. Ths wll allow for more samples to be traned. After the mages are acqured, they are segmented to provde a normalzed rectangular sample of the rs. Features are extracted usng 2-D mult-level wavelet transforms. For ths experment, 3 levels are used producng a total of 12 parts. The 12 parts produce 12 feature vectors consstng of the coeffcents from the wavelet transform. The mean and varance of each vector are obtaned to produce a total of 24 features for each sample. See [16] for more nformaton on the 2-D wavelet transforms used. 3.2 Classfers In recent years, the SVM classfer has ganed consderable popularty among the possble classfers. The objectve of the SVM classfer s to separate data wth a maxmal margn, whch tends to result n a better generalzaton of the data. Generalzaton helps wth the common classfcaton problem of over-fttng. The ponts that le on the planes that separate the data are the support vectors. Fndng the support vectors requres solvng the followng optmzaton problem (de-tals of ths method can be found n [2, 13]): mn 1 2 w, b, ξ T subject to: y ( w φ( x ) + b) 1 w l w + C ξ = 1 ξ, ξ > 0 The geometrc representaton of the SVM s easly vsualzed when the data falls nto the lnear separable case. It becomes more complex as the data falls nto lnear nonseparable case and non-lnear separable cases. For more nformaton on the dfferent cases, please refer to [11] whch devotes a chapter on SVMs. Real world data tends to fall nto the non-lnear separable case. To solve the non-lnear separable problem, the SVM T (2) reles on pre-processng the data to represent patterns n a hgher dmenson than the orgnal data set. The functons that provde the mappng to hgher dmensons are known as ph functons or kernels. Common kernels nclude Radal Bass Functon (RBF), lnear, polynomal, and sgmod. The RBF kernel s used n ths study and addtonal nformaton on ths kernel follows n secton 4. The other classfer we consder s the Nearest Neghbor classfer, whch computes dstances from a test subject d to each member d of the database, and classfes the test subject as the subject that has the closest dstance. The dstances can be computed usng varous methods such cty-block dstance or Eucldean dstance. A reject threshold can be ntroduced nto the Nearest Neghbor classfcaton. If the dstance between test subject d and ts nearest neghbor d s wthn the threshold, the classfcaton s that of the closest member. However, f the dstance s greater than the threshold, the subject s rejected and classfed as a non-member. In ths study, we used a reject threshold. 4 EXPERIMENTS Our hypothess s that bometrc dentfcaton on closed systems does not generalze well to larger, open systems contanng non-members. In order to nvestgate ths hypothess, experments were conducted on subset database M ' M from both the wrter and rs databases descrbed n secton 3. 4.1 Experment Setup For each of the databases, tranng sets were created. Tranng sets for the wrter data conssted of m ' = 50, 100, 200 and 400 members. Tranng sets for the rs data conssted of m ' = 5, 15, 25 and 35 members. These sets ncluded all nstances per member,.e., 3 per member for wrter and 10 per member for rs. For the frst part of the experment, an SVM was traned on the members. Parameter tunng, or SVM optmzaton, was performed pror to tranng. The frst parameter tuned s the penalty parameter C from equaton (2), and dependng on the kernel used, there are addtonal parameters to tune. For ths experment we used an RBF kernel of the form: K( x, xj) = γ x xj, γ > 0 e The γ parameter n equaton (3) s the only kernel parameter requrng tunng. A grd-search method as defned n [8] was used to optmze these two parameters. Tunng the parameters gves 100% accuracy on each of the tranng sets. Therefore, we have 0% FAR and FRR, or equvalently, 100% securty and convenence. The next 2 (3) D5.3
step s to test non-members to determne the true securty of the traned SVM. For each tranng set we created a combned evaluaton sets consstng of the traned members plus an ncreasng number of non-members. The evaluaton sets for the 50-wrter traned SVM conssted of 50, 100, 200, 400, 700 and 841 subjects, where the frst 50 subjects are the members and the remanng subjects are non-members. Smlarly, the evaluaton sets for the 25-rs traned SVM conssted of 25, 35, 45 and 52 subjects, where the frst 25 subjects are the members and remanng subjects are non-members. 4.2 Results and Analyss In the postve dentfcaton model we consder two errors: false accepts and false rejects. Snce the SVM was able to tran the members to 100% accuracy, we elmnate the false accepts and false rejects for members. The remanng tests are non-members and therefore can only produce false accepts. The false accepts correlate to the securty measurement of the system, a measure of extreme mportance to the system. In fgure 3, the securty results are shown for the wrter data. In the second part of the experment, the Nearest Neghbor classfer was used. For ths classfer, threshold tunng was requred. The threshold has to be large enough to allow dentfcaton for the known members, but small enough not to allow non-members to be classfed as members. As the threshold ncreases, the FAR ncreases and FRR decreases. The Recever Operatng Characterstc (ROC) curve for the Nearest Neghbor classfer s presented n fgure 2. The ROC curve s a plot of FAR aganst FRR for varous thresholds. When desgnng an dentfcaton system, there s a trade off between the convenence (FRR) and securty (FAR) of the system. For ths experment, we have chosen an operatng threshold that s close to equal error rate, but leanng towards a hgher securty system. Fgure 3. Securty results for wrter data usng SVM Fgure 4. Securty results for rs data usng SVM. Fgure 2. ROC curve for Nearest Neghbor wth 100 members. As hypotheszed, for each curve, as the number of nonmembers ncreases, the securty monotoncally decreases (or equvalently, the FAR monotoncally ncreases). It mght also be noted that the fnal rates to whch the securty curves decrease appear to converge that s, to approach asymptotes. To ensure that ths s not an artfact of the partcular handwrtng data used, we obtaned smlar experment results on the rs data as presented n fgure 4. The rs data n fgure 4 follows the same pattern as the wrter data n fgure 3, although convergence s not as evdent for these data. D5.4
Next, we present the results for the Nearest Neghbor classfer. As can be seen n fgure 5, the same pattern emerges, although n ths experment we dd not obtan 0% FAR for the members. When usng the Nearest Neghbor approach, a one-versus-all method was used to obtan the accuracy for the closed envronment. Last, we go one step further brngng n an addtonal classfer, the Artfcal Neural Network or ANN. Fgure 7 llustrates the securty performance for 15 members of the rs database. We notce the Nearest Neghbor agan does not perform well on the closed envronment, but does out perform the SVM performance as non-members enter the system. However, we ntroduced a 1-versus-all approach ANN. Ths approach, lke SVM, provdes excellent performance on the closed envronment, and actually surpasses both the SVM and NN on the open envronment. Ths s an nterestng fnd, however, addtonal work stll needs to performed to test aganst varous szed rs and wrter data. Fgure 5. Securty results for wrter data usng Nearest Neghbor. We now present a comparson of the results from the two classfers used n ths experment. Fgure 6 llustrates the securty performance for 100 members of the wrter database. Notce, although Nearest Neghbor does not perform as well on the closed envronment, t eventually meets and surpasses the performance of the SVM as non-members enter the system. Fgure 7. A comparson of the performance of the Nearest Neghbor, SVM and ANN classfers on the rs data of 15 members. 4.3 Securty Convergence Based on the securty results of fgures 3, 4 and 5, we recognze that the curves appear to be of exponental form and that we mght be able to extrapolate the securty of a system for large populatons contanng non-members. After some fttng trals, we fnd the curve most smlar to be y 1 (( x b) / c) 2 = + ae where the constant b s the number of members; the constants a, c, and d vary based on b; and the securty converges to d. Fgure 8 dsplays the curve fttng for the 50 and 200 traned wrter data samples. d (4) Fgure 6. A comparson of the performance of the Nearest Neghbor and SVM classfers on the wrter data of 100 members. D5.5
In summary, we demonstrated that the generalzaton capablty of closed bometrc systems n open envronments s poor, and that the sgnfcantly larger error rates should be taken nto account when desgnng bometrc systems for postve dentfcaton. For ncreased securty, for example, mult-modal bometrcs mght be consdered [14]. 5.1 Future work When desgnng an dentfcaton system, there s a trade off between the convenence and securty of the system. Most systems would choose securty over convenence. However, n our mplementaton of SVM for the wrter data, we mply choosng convenence over securty (guarantee 0 false rejects). In our study on wrter data, there just are not enough samples per member to put nto the testng set. It would therefore be benefcal to run further experments aganst larger bometrc databases. Fgure 8. Curve fttng for wrter data of 50 and 200 members. 5 CONCLUSIONS In ths paper, we found that system securty (1-FAR) decreases rapdly for closed systems when they are tested n open-system mode as the number of non members tested ncreases. Thus, the hgh accuracy rates often obtaned for closed bometrc dentfcaton problems do not appear to generalze well to the open system problem. Ths s mportant because we beleve that no system can be guaranteed to reman closed. Ths hypothess was valdated by experments on both wrter and rs bometrc databases. An estmate of the expected error was also projected based on the asymptote of an exponental curve ftted to the data. Furthermore, because the FAR n these studes was obtaned from normal bometrcs of other subjects, the socalled "zero-effort" rate, the true FAR of these systems would be greater, resultng n even poorer generalzaton. We also found that, although systems can be traned for greater closed-system securty usng SVM rather than NN classfers, the NN systems are better for generalzng to open systems through ther capablty of rejectng non members. Thus, t appears that the reject thresholds of NN classfers do a better job of rejectng non members than the reject regons of SVM classfers. Also, the ANN classfer may provde even better results when generalzng to open systems, but more work stll needs to be performed wth that classfer. Note, most complex bometrcs systems use more complex classfers. Gven that performance on closed systems s not suffcent for applcatons where securty s essental, we feel even the most complex classfers should be tested n an open envronment. For a more n depth analyss of varous classfers on open envronments, refer to [6] We thnk that t may be advantageous to develop open dentfcaton systems by usng a verfcaton model. Also, t would be benefcal to explore the features for the gven bometrcs snce securty results could mprove wth features that better dentfy a member. Addtonal forms of bometrcs, such as fngerprnt, face, voce, hand geometry or a combnaton of bometrcs mght also be tested to further test the hypothess. REFERENCES 1. Bolle, R.M., Connell, J.H., Pankant, S., Ratha, N.K., Senor, A.W.: Gude to Bometrcs. Sprnger (2004) 2. Burges, C.: A Tutoral on Support Vector Machnes for Pattern Recognton. Data Mnng and Knowledge Dscovery. 2:121-167 (1998) 3. Cha, S.-H., Srhar, S.N.: Wrter Identfcaton: Statstcal Analyss and Dchotomzer. Proceedngs Internatonal Workshop on Structural and Syntactcal Pattern Recognton (SSPR 2000), Alcante, Span. pp. 123-132 (2000) 4. Cha, S.-H., Srhar, S.N.: Handwrtten Document Image Database Constructon and Retreval System. Proceedngs of SPIE Document Recognton and Retreval VIII, Vol. 4307 San Jose (2001) 5. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classfcaton. John Wley & Sons (2001) 6. Gbbons, M.: On Evaluatng Open Bometrc Identfcaton Systems. Master s Thess, CSIS Department, Pace Unversty (2005) 7. Grother, P., Phllps, P.J.: Models of Large Populaton Recognton Performance. 2004 IEEE Computer Socety Conference on Computer Vson and Pattern Recognton (CVPR'04) Vol. 2. Washngton, D.C. pp. 68-77 (2004) 8. Hsu, C.-W., Chang, C.-C., Len, C.-J.: A Practcal Gude to Support Vector Classfcaton. 9. Kee, G., Byun, Y., Lee, K., Lee, Y.: Improved Technques for an Irs Recognton System wth Hgh Performance. Lecture Notes Artfcal Intellgence (2001) 10. Krchen, E., Mellakh, M.A., Garca-Salcett, S., Dorzz, B.: Irs Identfcaton Usng Wavelet Packets. Pattern Recognton, 17th Internatonal Conference on (ICPR'04) Vol. 4. pp. 335-338 (2004) D5.6
11. Kung, S.Y., Mak, M.W., Ln, S.H.: Bometrc Authentcaton: A Machne Learnng Ap-proach. Pearson Educaton Company (2004) 12. Ma, Y., Pollck, F., Hewtt, W.T.: Usng B-Splne Curves for Hand Recognton. Pattern Recognton, 17th Internatonal Conference on (ICPR'04) Vol. 3. Cambrdge UK. pp. 274-277 (2004) 13. Osuna, E., Freund, R., Gros, F.: Support Vector Machnes: Tranng and Applcatons. MIT Artfcal Intellgence Laboratory and Center for Bologcal and Computatonal Learn-ng Department of Bran and Cogntve Scences. A.I. Memo No 1602, C.B.C.L. Paper No 144 (1997) and Machne Intellgence. Vol. 25, No. 9. pp. 1041-1050 (2003) 14. Prabhakar, S., Pankant, S., Jan, A.K.: Bometrc Recognton: Securty & Prvacy Con-cerns. IEEE Securty and Prvacy Magazne. Vol. 1, No. 2. pp. 33-42 (2003) 15. Schlapbach, A., Bunke, H.: Off-lne Handwrtng Identfcaton Usng HMM Based Recog-nzers. Pattern Recognton, 17th Internatonal Conference on (ICPR'04) Vol. 2. Cambrdge UK. pp. 654-658 (2004) 16. Woodford, B.J., Deng, D., Benwell, G.L.: A Wavelet-based Neuro-fuzzy System for Data Mnng Small Image Sets. 17. Zhang, D., Kong, W.-K., You, J., Wong, M.: Onlne Palmprnt Identfcaton. IEEE Trans-actons on Pattern Analyss D5.7