INTERPRETING FINGERPRINT AUTHENTICATION PERFORMANCE TECHNICAL WHITE PAPER

Transcription

1 INTERPRETING FINGERPRINT AUTHENTICATION PERFORMANCE TECHNICAL WHITE PAPER Fidelica Microsystems, Inc. 423 Dixon Landing Rd. Milpitas, CA

2 INTRODUCTION The fingerprint-imaging segment of the biometrics industry is in its embryonic state; a myriad of technologies and concepts have fostered slow but steady growth. As the number of applications and level of acceptance grow, biometric developers are faced with the task of evaluating and contrasting the various solutions on the market. To date, developers of fingerprint authentication products, such as sensors and algorithms, have developed tests tailored to their particular devices and software. Since the relative performance of the fingerprint authentication systems is crucial in the evaluation process, it is imperative that the results be analyzed with care. A comprehensive discussion of performance results is the subject of this white paper. The focus is on the entire authentication system, including the fingerprint image sensor and verification algorithm. In the process, some common definitions and test procedure pitfalls are discussed. PERFORMANCE METRICS Performance tests of fingerprint authentication systems come in many varieties. Some tests are done in the laboratory; others are performed in the field. In addition, these systems usually can be operated under different security settings producing a spread of performance results. There are many different variables to be aware of when testing performance or analyzing test results provided by different vendors. Using any authentication system requires two steps: enrollment and verification. For a passwordbased authentication system, the enrollment is simply the creation of a password. Often times, the enrollment requires the user to re-enter the password to assure that the stored password is what the user intended. Verification in such an authentication system would comprise nothing more than the submission of a password in order to gain access to restricted areas. Similarly, for a fingerprint authentication system, the enrollment step would consist of the user submitting fingerprint image samples to the system. The system would then analyze the samples and generate a template to be stored. Once a user has been enrolled in the system, verification of the user can take place. This would involve the user submitting a fingerprint image sample to be compared with the enrolled template. If the newly submitted image sample matches the stored template, the user is accepted. Otherwise, the user is rejected. Sometimes, the system s decision to reject or accept a user is erroneous. How frequently systems make an incorrect decision is of great importance. There are four basic error rates employed in fingerprint authentication systems: the False Accept Rate (FAR); the False Reject Rate (FRR); the Equal Error Rate (EER); the Failure to Enroll Rate (FER). The FAR represents the probability that a false match occurs, for example, an unauthorized user is erroneously accepted as an authorized user by the system. Its complement is the FRR, which represents the probability that a false rejection occurs, for example, an authorized user is 2

3 erroneously mistaken for an unauthorized user. The FER represents the probability that a single fingerprint cannot be enrolled in the system, such as when an individual wants to use his right thumb with the fingerprint authentication system, but for some reason the system determines that his right thumb is not usable for this purpose. Before the EER can be described, some additional background information must be provided about the performance of these systems. First, understand that there is a trade-off between the FAR and FRR. If a system has various security settings at which it can be operated, then the FAR and FRR will vary in accordance with the security setting selected. In general, as the FAR is reduced the system becomes more secure the FRR is increased. A consequence of a higher FRR is user inconvenience, since successful authentication of an authorized user may require additional access attempts. For a given system, there may be only a handful of security settings available, but the error rates from an infinite number of these settings can be interpolated to form a continuous FAR/FRR trade-off curve. The trade-off curve most often associated with fingerprint authentication systems is known as a Receiver Operating Characteristics (ROC) curve. The ROC curve plots the FAR against the FRR. It can be generated by obtaining FAR and FRR data points under many different security settings on a given system (sensor and verification algorithm). Fig. 1 shows an example of a ROC curve. From this curve, one can easily see the trade-off between FAR and FRR. Fig. 1. With a basic understanding of ROC curves and the FAR-FRR trade-off, the EER can now be described. The EER is the datum on the ROC curve where the FAR is equal to the FRR. In the ROC curve shown in Fig. 1, the EER is approximately The EER is often used for comparisons because it is simpler to obtain and compare a single value characterizing the system performance, as opposed to generating and using full ROC curves. Unfortunately, the EER is 3

4 rarely a good choice for comparing the performance of various fingerprint authentication systems. In particular, the efficacy of a given fingerprint authentication system is highly dependent on its intended application. Some applications require higher security and can sacrifice some convenience, but in others convenience is a top priority. This is the main problem with using the EER as a basis for comparing system performance. Since the EER only gives information that is useful for a single application requirement, it will vary from system to system. Therefore, EERs often compare systems developed for potentially different applications. To further clarify this point, let us consider an example. Say that a company is evaluating several fingerprint authentication systems with the intention of deployment in a high security environment. The company is willing to accept a higher FRR in order to ensure that the chances an unauthorized user can gain access is minimal (for example, an FAR of 1 in 250,000). If the systems being evaluated had the EERs given in Table I, none would clearly stand out as the best because nothing would be known about them for the intended high security application. From this example, one can quickly see the limitations of the EER. Table I. An illustration of the limitations of the EER. Fingerprint Authentication System EER Fingerprint System A 0.2% Fingerprint System B 0.7% Fingerprint System C 1.0% Similar problems plague the FAR and FRR numbers often reported by fingerprint authentication algorithm vendors. With a single reported FAR and FRR, the performance cannot be properly assessed for a given application. This is why it is necessary to compare systems using a family of ROC curves. ROC curves allow for the comparison of various authentication systems for a given application. It is not unusual for one system to outperform another in one application and have their roles reversed in another. This scenario would manifest itself as a crossover in the ROC curves as seen in Fig. 2. These ROC curves show one system performing better than the other in a lower security (higher FAR) environment and worse than the other in a higher security (lower FAR) environment. 4

5 Fig. 2. TESTING PITFALLS Having described some of the fundamental issues in comparing the performance of fingerprint authentication systems, our attention can now be shifted to actual test practices. It is important to understand exactly how the FAR and FRR that make up the ROC curves are determined. Unfortunately, a widely accepted biometrics testing standard currently does not exist. Because of this, it is easy for a specific vendor to skew test results to benefit their own system. In fact, results from fingerprint authentication tests will undoubtedly be biased by a number of uncontrollable factors. The performance results from the test of a fingerprint authentication system are largely affected by the test population. Error rates gathered from a test population containing more individuals with poor quality fingerprints are likely to be worse than a similar test with a population containing more individuals with much higher quality fingerprints. Population selection is highly unscientific. Often times the population is limited to those who volunteer their time. Little or no control of the population is a common occurrence in the testing of authentication systems. Typically, the reported FRR and FAR are determined by the number of failing fingerprint match trials and the number of failing non-match trials, respectively. Say the test to determine the FRR of the system for a given security setting consisted of 20,000 match trials. The FRR is determined by dividing the number of trials that failed to produce the correct result (true match) by the total number of trials. Suppose that of the 20,000 match trials, 250 of them resulted in rejected access. Then the FRR for this example is 250 / 20,000 =.0125 or 1.25%. A similar test could be used to determine the FAR of the system. However, since false accepts occur a lot less frequently than FRRs, many more trials must be run in order to obtain a reasonably accurate FAR. 5

6 There is a mathematical rule of thumb known as Doddington s 30-error criterion, which states that for a measured error rate to lie within 25% of the true error rate with high confidence (90%) then at least 30 instances of that error should have been observed. For example, one can t put too much confidence in a measured FAR rate for some system at a specified security setting of 1 in 100,000 if only 100,000 trials were performed. This is because only one false accept was observed after 100,000 trials. However, if the measured FAR was 1 in 100,000 after 3,000,000 trials, then it can be said that the true error rate is within 25% of the observed error rate with a much higher degree of confidence. In this particular case, the FAR can be said to be 0.001% ± % with 90% confidence based on the Doddington 30-error criterion. Thus, with this rule in mind, one can get a sense of the accuracy of reported error rates from various system vendors once the number of matching trials is known. The independence of trials is another topic to consider when performing due diligence on vendors error rate claims. Ideally, a test of a fingerprint authentication system should consist of what are known as Bernoulli trials. A Bernoulli trial is one that returns a binary output (in the case of a fingerprint authentication system, either a MATCH or a NO MATCH) and is statistically independent from other trials. This is not feasible for large tests, because as soon as an single individual s fingerprint is used in more than a single trial, the independence requirement of a Bernoulli trial is violated. This is because each individual fingerprint s has an error probability distribution with respect to those of another individual. As an example, say Individual A s fingerprints are more likely to match against those of Individual B. Then by having multiple trials of matching fingerprint images from Individuals A and B, the FAR will be inflated. Finally, careful attention must be paid to other factors that could affect test results. One must understand that sources of testing bias cannot be completely eliminated. They are a part of any test involving fingerprint authentication systems. However, it is important to be aware of any bias that might affect the test results. When comparing error rates published by different vendors, one should be conscious of the differences between test parameters and the definitions of the various errors. For example, one vendor might allow three retries in a given trial before considering the trial to be a single failed attempt to match; another vendor might count this as three failed attempts to match. Some consider the FRR to be a measure of false rejects of those who could be enrolled in the system; others consider any failed enrollment to be a contributor to the FRR. Therefore, it is important to have a strong grasp of what the precise definitions of the error rates are when making comparisons of ROC curves. In the absence of published testing standards, this is a very real issue. 6