How accurate is AGNITIO KIVOX Voice ID?

Transcription

1 How accurate is AGNITIO KIVOX Voice ID? Overview Using natural speech, KIVOX can work with error rates below 1%. When optimized for short utterances, where the same phrase is used for enrolment and authentication, KIVOX can work with error rates as low as 0.5%. Error rates depend on several factors, such as length of utterance, audio quality and working points. To predict the error rates for a specific use case, audio samples that are as close as possible to the real world samples will be needed, and the threshold (high security or high convenience) in the desired working point will need to be set appropriately. KIVOX can achieve error rates below 1% for complete free speech conversations of seven seconds in length in a typical call center environment, using a voice print created from at least 30 seconds of speech. KIVOX can be optimized for short utterances in collaborative situations to obtain better error rates with shorter utterances. In that case, the user will enroll by repeating a selected sentence, of two to three seconds in length, three times and then they can authenticate by repeating the same sentence once. When a user is always authenticating via the same device, like a smart phone, or tablet, error rates below 0.5% are achieved. These error rates change to 1%-5% when very short phrases of less than two seconds are used, such as those used for wake-up functions. Scenarios where more challenging telephonic infrastructures are used (like call center or IVR environments) and customers can use different methods for authentication, can also result in these slightly higher rates. 1. Accuracy metrics for biometric authentication A standard authentication system usually returns a confidence score that determines whether or not the identity of the user is confirmed and they are successfully authenticated. That score can come from many different sources. For AGNITIO products, this score comes from a comparison made between the existing voice print of the user and the new access sample that is taken during the authentication process. This likelihood score is typically higher as the new sample is likely to be more similar to the already existing voice print. In order to make a decision, based on that score, the most typical approach is to set a threshold. Every time the score is above that set threshold, the user is verified, and if the score is below, the user is rejected. Using this method, two types of error can occur: 1 of 5

2 False rejection error: when a legitimate user is rejected by the authentication system (also referred to as a missed probability) False positive error: when an impostor is erroneously accepted as a legitimate user (also referred to as a false acceptance (FA) error) These two types of errors are related to the set threshold. When the threshold is set higher, false acceptance errors will be reduced, but the probability of a false rejection (FR) will be higher. The opposite behaviour will be obtained moving the existing threshold to a lower value (reducing the false rejections but increasing the false acceptances). So, we can conclude that FR and FA are inversely proportional. For this reason, an infinite number of working points are possible, which are the specific pairs of FR/FA error rates selected for a given platform and threshold. In order to measure and compare the accuracy of different systems, it is not possible to use a single working point, because depending on the specific application or use case, a different setting will be needed. There is therefore a need to understand the behaviour of that authentication technology in all possible working points. In order to do that, the DET (Detection Error Trade-off) curves are used. The idea is try to calculate all the possible FR/FA pairs for a given system by moving back and forth through the thresholds. All the working points obtained are then represented as a single curve where the X and Y axis are representing false acceptance rates (FAR) and false rejection rates (FRR) respectively. There is a special working point that is often used in order to compare two systems with a single number. That point is the Equal Error Rate (EER). The EER point has the property of having FAR and FRR equal. The EER is easy to obtain as the crossing point between the DET curve and the bisecting line of the graph. In general terms, DET curves within systems are parallel, so that means that the system with the lower EER is usually the one performing the best across all the working points. EER is a good metric to summarize the accuracy of a biometric system in a single number, but, if a system has an EER equal to 1%, it doesn t mean that you cannot achieve lower FAR or FRR, it simply means that if you want to get an FAR lower than 1%, you will need to have an FRR higher than 1% and vice versa. Therefore EER represents the point where you can achieve the lowest error rates at the same time. Usually, system don t use the EER as the working point. High security systems require very low FA (below EER) so they will have a higher FRR than the EER. In the same way, a user friendly system requires low FR (lower than the EER) making the FA higher. These points are illustrated in the following DET graphic: 2 of 5

3 2. AGNITIO s KIVOX Accuracy AGNITIO technology is based on natural speech. In collaborative environments, where the same passphrase is used for enrolment and authentication, better accuracy can be achieved for very short utterances. KIVOX can therefore authenticate users in two different modes. Firstly, when the user is actively participating in the authentication process via an app or an IVR system, a text dependent (phrase-specific) approach is used in order to reduce the time for the authentication. Secondly, when the authentication is done in a non-collaborative/transparent way, when the user is speaking freely with an agent or using the navigation commands of a smartphone, a text independent (natural speech) approach is used. 3. KIVOX Passphrase option (Text dependent) In this case, the user enrolls by repeating a two to three second long sentence, three times. Once the user models are created, only a single repetition of that same phrase is needed to authenticate the user. From there, and depending on the specific use case, KIVOX will obtain a variety of different error rates. For those scenarios where KIVOX is used in the Cloud to get users authenticated while using a unique device (typically a smartphone or tablet where are accessing protected content or an app), KIVOX s proprietary text dependent technology performs with EER as low as 0.5% depending on the sentence selected. Below, we see some DET curves that are evaluating the performance of different phrases: This level of performance means that the system can operate with working points as low as FA=FR=0.5%. Lower levels of FA or FR are possible if the user will accept a higher level of error. While two to three seconds long is the standard length for an authentication phrase, a wider variety of use cases can be tackled with KIVOX server based technology. The most typical of these use cases is when the technology is embedded in a call center/ IVR infrastructure. In that kind of environment, a wide variety of elements (switches, CTI, IVR, trunks) can affect the quality of audio. In addition, in many cases, the user will want to authenticate from different devices (home landline phone, mobile phone, office VoIP phone, etc.). In these scenarios, KIVOX is a powerful tool as it allows enrolment and authentication to be performed across many different kinds of communication 3 of 5

4 channel. In these situations, it is quite normal to obtain lower accuracy numbers, but AGNITIO KIVOX technology can work with an EER of 1% to 5%, even in the most challenging environments and infrastructures. Below you can find a couple of examples of performance obtained in real deployments: It can be observed that even in the most challenging scenarios (right graph), it is possible to get False Alarm rates as low as 0.5% while still successfully authenticating more than 90% of your legitimate users, and that can increase to 95-98% in better environments. 4. KIVOX Natural Speech Option (Text independent) For natural speech authentication, a minimum of 30 seconds of free speech from the user is needed to enroll. As this environment is fully text independent, an old recorded conversation of the user with an agent in a call center, or historical data saved from the user navigating through an automated system, can also be used. Once enrolled, only seven seconds of speech is needed for authentication. For these reasons, it is this type of technology that is preferred in a call center environment because even with this more complex scenario, the levels of accuracy can be kept very high. In optimal conditions, KIVOX can run in a fully text independent environment with an EER below 1%. One could imagine that error rates using text independent authentication would be higher because in text dependent mode, you have a very restrictive vocabulary that makes authentication easier. However, the more speech samples available for a specific user, the better the accuracy of the system, because the biometric voice print contains a much richer, more variable collection of that person s speaking voice. The typical performance of a system with a number of different length speech samples used for authentication is shown below. Again, we must take into consideration that it is possible to obtain working points where FA or FR can be lower than 1% and we typically see variation in the range of 1% to 3%. The chart on the next page shows performance based on the length of audio used to verify the speaker (20, 30 and 60 seconds). In all cases, five calls from the specific customer were used to initially train the model. As you can see, in the 60 second examples, a working point with FA of 0.1% and FR of 2.5% can be achieved. 4 of 5

5 About AGNITIO AGNITIO is a recognized global leader in voice biometrics, with highly accurate, spoof-protected, ultra-fast technologies for speaker identification. The company works with partners worldwide to enable voice identification in the most challenging environments, including intelligence agencies, law enforcement and corporations. AGNITIO is now pioneering voice identification solutions for online transactions and business applications. The AGNITIO voice biometrics engine offers a high degree of accuracy, with the added protection of patented anti-spoofing techniques. The flexible voice identification engine, available to partners through an API, works in a wide range of applications and across languages. For more information or to request a demonstration, visit 5 of AGNITIO