A Misclassification Reduction Approach for Automatic Call Routing

Transcription

1 A Msclassfcaton Reducton Approach for Automatc Call Routng Fernando Uceda-Ponga 1, Lus Vllaseñor-Pneda 1, Manuel Montes-y-Gómez 1, Alejandro Barbosa 2 1 Laboratoro de Tecnologías del Lenguaje, INAOE, Méxco. {fuceda, vllasen, mmontesg}@naoep.mx 2 Nuance Technologes, Mexco. Alejandro.Barbosa@nuance.com Abstract. Automatc call routng s one of the most mportant ssues n the call center doman. It can be modeled once performed the speech recognton of utterances as a text classfcaton task. Nevertheless, n ths case, texts are extremely small (just a few words) and there are a great number of narrow call-type classes. In ths paper, we propose a text classfcaton method specally suted to work on ths scenaro. Ths method consders a new weghtng scheme of terms and uses a multple stage classfcaton approach wth the am of balance the rate of rejected calls (drected to a human operator) and the classfcaton accuracy. The proposed method was evaluated on a Spansh corpus consstng of 24,638 call utterances achevng outstandng results: 95.5% of classfcaton accuracy wth a rejecton rate of just 8.2%. 1 Introducton Call routng (CR) s one of the most mportant ssues n the call center doman. It can be defned as the process of assocatng user requests wth ther desre destnatons [1]. In other words, t nvolves forwardng ncomng phone calls to someone qualfed to handle them (e.g., to the most approprate customer representatve). Tradtonally, ths task has been performed usng a touch-tone approach, whch allows users to navgate through dfferent optons by a herarchcal menu. In spte of ts conceptual smplcty, ths approach has several dsadvantages, such as the tme the users need for lstenng all optons as well as the dffculty for matchng exstng optons wth ther specfc needs. These nconvenences are evdent even for smple requests such as I want to know my account balance, whch may requre users to navgate as many as four or fve nested menus wth four or fve optons each [2]. The nconvenences of the tone-touch approach leaded to the emergence of automatc call routng systems, whch allow users to nteract n natural spoken language [1], and therefore, help companes to sgnfcantly reduce ther telephone tme costs. Typcally, ths knd of systems consders two man steps. In the frst one, phone calls are transcrbed to text by means of a speech recognton process. Then, n a second step, text transcrptons are used to predct the correct destnaton for the phone calls.

2 In partcular, ths paper focuses on the second step, whch, n fact, can be consdered as a text classfcaton problem. Text classfcaton, the assgnment of free text documents to one or more predefned categores based on ther content, s a wdely studed problem that has recently acheved sgnfcant advances [3]. Nevertheless, t s mportant to pont out that automatc call routng s a more dffcult problem than ordnary text classfcaton, snce phone call transcrptons are very short and nosy texts. That s, they are texts consstng of just a few words and contanng many errors such as word nsertons, deletons and substtutons. In addton, automatc call routng dffers from conventonal text classfcaton n that, n the former, there s always the possblty of appealng for human assstance. In partcular, call routng systems tend to reject all ambguous or ncomprehensble transcrptons, transferrng ther resoluton to a human operator. Evdently, ths crcumstance s of great relevance, snce havng a hgh rejecton rate mples a hgh telephone tme cost, whereas havng a low classfcaton accuracy (due to the msclassfcaton of unclear transcrptons) causes a strong dscomfort among users. Therefore, one of the man research topcs on automatc call routng s to provde a good balance between these two factors, the classfcaton accuracy and the rejecton rate. In ths paper, we propose a classfcaton method specally suted to the task of automatc call routng. On the one hand, ths method consders a new weghtng scheme that s especally approprate for short texts. On the other hand, t uses a twostep classfcaton approach that allows achevng an adequate balance between the rejecton rate and the classfcaton accuracy. In a frst step, ths approach guarantees hgh classfcaton accuracy by toleratng a hgh rejecton rate. Subsequently, n a second step, t reclassfy rejected transcrptons wth the ntenton of recoverng some nstances and, therefore, reducng the fnal rejecton rate. Expermental results on a Spansh corpus consstng of 24,638 call utterances are encouragng; the proposed method could correctly classfed 95.5% of the utterances wth a rejecton rate of just 8.2%, outperformng the applcaton of other text classfcaton approaches. The rest of the paper s organzed as follows. Secton 2 descrbes some prevous work on automatc call routng. Secton 3 descrbes the proposed method for automatc call routng. Secton 4 presents the evaluaton results. Fnally, Secton 5 gves our conclusons and descrbes some deas for future work. 2 Related work There exst dfferent approaches for automatc call routng; among them we can menton the followng three ones: 1. A straghtforward approach based on the use of keywords for trggerng dfferent destnatons [4]. 2. An approach based on the use of language models that descrbe the man characterstcs (words sequences) of each destnaton [5]. 3. An approach based on the applcaton of tradtonal text classfcaton methods [2,6].

3 From these approaches, the frst one s the smplest to understand and mplement; nevertheless, t s very sensble to transcrpton errors. The second approach s more robust than the former, but t requres more tranng data as well as clearly dfferentable destnaton vocabulares. Fnally, the thrd approach s less sensble to transcrpton errors, but t s damaged by the short length of transcrptons as well as by the great number of narrow destnatons (categores). Dfferent works have evaluated the effectveness of conventonal text classfcaton methods for automatc call routng. For nstance, [7] shows a comparson of several learnng algorthms on a corpus of 4000 manual transcrptons correspondng to nne balanced categores. It reports a result of 81.6% of accuracy usng a smple recurrent network. In the same way, [8] reports an accuracy of 75.9%; nevertheless, n ths case they used a small corpus of 743 automatc transcrptons from sx dfferent categores. Both works do not report the rejecton rate. More recently, some works have proposed dfferent adaptatons to the tradtonal text classfcaton approach. These adaptatons consder the use of dfferent utterance characterzatons, weghtng schemes, feature selecton methods and learnng strateges. For nstance, [9] presents a learnng approach that uses a cascade of bnary classfers (of Support Vector Machnes) wth the am of reducng the classfcaton errors. In ths approach, those nstances that could not be classfed (.e., that pass through all classfers) are rejected. Reported results on a set of automatc transcrptons from 19 categores are very relevant; on the one hand, t reports a best accuracy of 99% correspondng to a rejecton rate of 75%, and on the other hand, a best rejecton rate of 15% for a classfcaton accuracy of 95%. Smlar to these works, our automatc call routng method s also based on a supervsed learnng approach. Its man dfference relays on the use of a novel two-step classfcaton approach that allows mantanng a good balance between the accuracy and rejecton rates. In addton, t apples a weghtng scheme that s less sensble to the short length of texts. The followng secton ntroduces the proposed method. 3 Proposed Method As we prevously mentoned, our method ncludes two major modfcatons to the tradtonal text classfcaton approach. On the one hand, t uses a new term weghtng scheme that allows mprovng the dscrmnaton among narrow classes formed by very short texts. On the other hand, t consders a two-step classfcaton approach that helps mantanng an adequate balance between the accuracy and rejecton rates. In the followng subsectons, we descrbe n detal these two modfcatons. 3.1 Term Weghtng Scheme Typcally, term relevance s determned usng the well-known tf-df weghtng scheme. Ths scheme, however, presents some drawbacks for ts applcaton to automatc call routng. Frst, gven the short length of texts, the tf-value s qute smlar

4 for most of the terms from a category 1. Second, gven the presence of several related narrow categores, the df-value s also very smlar for most terms. Therefore, ths weghtng scheme does not allow achevng a good dscrmnaton among categores. Based on the prevous observatons, we propose a weghtng scheme that computes the weght of terms n relaton to ther occurrence n the whole categores nstead than n ndvdual utterances. Ths scheme consders that the probablty of occurrence of a term across dstnct categores s dfferent. In partcular, we compute the weght of a term t n a category c C as follows: c c p t wt = c (1) max p p c t k c = P( c c ( ) k t) = c t ck ft c C where C s the set of categores, f t s the frequency of occurrence of the term t n c the category c, and pt ndcates the condtonal probablty P(c t), that s, the probablty of havng the category c gven the presence of the term t. It s nterestng to notce that the fnal weght of a term n a category s obtaned by applyng a knd of normalzaton wth respect to the probabltes of other terms from the same category. Roughly speakng, ths normalzaton allows gvng greater weght to terms clearly related to the category and reducng the value of terms unformly dstrbuted among several classes. 3.2 Two-Step Classfcaton Approach Fgure 1 shows the general archtecture of the proposed classfcaton approach, whch conssts of two man steps: a hgh-precson classfcaton step and a msclassfcaton reducton step. The man purpose of ths new approach s to acheve a good balance between the classfcaton accuracy and the rejecton rate. The frst step acheves an ntal classfcaton of the nput utterances (.e., the phone call transcrptons). Ths classfcaton consders all categores ncludng an unknown category that gathers all ambguous and ncomprehensble transcrptons. Ths category functons as a rejecton class, snce ts elements are transferred to a human operator for ther resoluton. Ths way, the purpose of ths step s to guarantee a hgh classfcaton accuracy by toleratng a hgh rejecton rate. In other words, n ths step only the most confdent utterances are sent to content categores, whereas the rest of them are classfed as unknown. Subsequently, n a second step, the dea s to reclassfy the set of rejected transcrptons wth the ntenton of recoverng some utterances and, therefore, reducng the fnal rejecton rate. Ths step combnes two addtonal classfers. One of them consders only two categores (unknown and the rest), and t s especally suted to dstngush clear ambguous and ncomprehensble transcrptons. The other one k f 1 Ths s because users tend to use a smple and drect language (more restrcted vocabulary) when they nteract wth a machne (an automatc system).

5 Hgh precson classfcaton step Input utterance Frst Classfer (n classes) Utterance classfed as Unknown NO Category of utterance YES Msclassfcaton reducton step Second Classfer (2 classes) Utterance classfed as Unknown NO Thrd Classfer (n 1 classes) YES Rejected utterance Fgure 1. General archtecture of the proposed method consders all categores except unknown and ts purpose s to reallocate rescue utterances wthn content categores. It s mportant to pont out that the weght of terms (defned n accordance to formula 1) vares from one classfer to another because they consder dfferent sets of categores. Ths characterstc of our approach s very relevant, snce t allows gvng the terms a convenent (hghly dscrmnatve) weght at each step. 4 Expermental Evaluaton 4.1 Tran and Test Corpora In order to evaluate the proposed method we used a corpus consstng of 24,638 Spansh automatc transcrptons 2. These transcrptons came from a company that provdes TV, nternet and telephone servces, and were manually classfed n 24 dfferent categores. The frst 23 categores correspond to content categores (.e., to possble destnatons for user requests), whereas the last one s an unknown category, whch gathers all utterances that could not be classfed and that must be rejected. For evaluaton purposes, we dvded the corpus n two non-overlapped subsets: a tran set formed by 80% of the nstances from each category (n total, 19,694 utterances), and a test set ncludng the remanng 20% of the nstances (4,944 utterances). Fgure 2 shows the dstrbuton of nstances per category; as t can be observed, ths s a very mbalanced corpus. 2 It was assembled by Nuance technologes ( usng the Wzard of Oz technque.

6 PaymentPlan TVOnDemand PhoneSupport PhoneChange VagueInternet ServceStart TelephoneStart SupportAtHome CableSuppo rt AccountB alance VagueCable CableStart ServceChange AccountInfo AddressChange TechSuport InternetSupport CableChange InternetStart Payments Retentons Representatve VaguePhone Unkno wn 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% Fgure 2. Percentage of nstances per category n the used corpus We decded usng ths evaluaton scheme because we wshed to smulate a real stuaton, where t s possble to fnd utterances contanng unspecfed words for the classfer. In partcular, the vocabulares from the tran and test sets showed mportant dfferences: 13.5% of the words from the test nstances were not ncluded n the tran set. 4.2 Results In order to obtan a general vson from the proposed method we performed several experments. Intally, we studed the usefulness of the weghtng scheme for tunng the accuracy and rejecton rates. Then, we evaluated the effectveness of our twostep classfcaton approach by comparng ts results aganst others from conventonal one-step classfers. Tunng the accuracy and rejecton rates For ths frst experment, we used the weght of terms as the man crteron for feature selecton. In partcular, we elmnated terms wth weght n all categores less than a gven specfed threshold. Table 1 presents the results correspondng to dfferent thresholds. It manly shows the number of tranng features, the classfcaton accuracy 3 as well as the rejecton rate 4. It s mportant to menton that n all cases we 3 Defned as the percentage of utterances correctly classfed wthn the set of 23 content categores. 4 Defned as the percentage of test utterances classfed as unknown.

7 used a sngle (one-step) mult-class Naïve Bayes classfer and appled a Boolean weghtng for the test utterances 5. Table 1. Feature selecton usng the proposed weghtng scheme Weght threshold Number of features Classfcaton accuracy Rejecton rate % 1.0% % 0.9% % 24.2% % 31.5% % 40.7% % 50.9% Results from ths ntal experment showed that usng dfferent thresholds t was possble to obtan dfferent combnatons of classfcaton accuraces and rejecton rates. As we expected, the use of the most dscrmnatve terms allowed achevng better classfcaton accuracy, but also ncremented the rejecton rate snce several nstances could not be represented. These results are of great relevance to our method snce t consders three dfferent classfers havng dfferent purposes. Evaluatng the effectveness of our two-step classfcaton method The goal of the proposed two-step classfcaton method s to acheve a good balance between the classfcaton accuracy and the rejecton rate. In order to evaluate the effectveness of ths method, we compared ts results aganst those from a conventonal one-step classfer (refer to Table 1). Our two-step classfcaton method, as shown n Fgure 1, consders the combnaton of three dfferent classfers. All of them, as n the prevous experment, were mplemented usng a Naïve Bayes classfer and Boolean weghts for representng test nstances. Nevertheless, n ths case, each classfer consders a dfferent weght threshold 6 : the frst two classfers used a threshold of 0.3, whereas the last one used a threshold of 0.2. Table 2. Results from the proposed two-step classfcaton method Correctly classfed Utterances Frst step Second step Classfcaton accuracy Rejecton rate % 8.2% Table 2 shows the results acheved by ths method. In ths case, the accuracy was calculated takng nto consderaton the correctly classfed nstances from the frst step as well as the correctly rescued nstances from the second step. On the other hand, the rejecton rate was obtaned from the set of nstances classfed as unknown 5 In contrast, tranng nstances were represented usng the proposed weghtng scheme. 6 These thresholds were emprcally determned; for each classfer we evaluated several thresholds and selected the combnaton that acheved the best overall accuracy.

8 by the second classfer. As can be observed, the acheved results are encouragng; the proposed method reached an adequate balance between the classfcaton accuracy and rejecton rate, better than the results from all prevous one-step classfers (refer to Table 1). Fnally, n order to have more nformaton to judge the effectveness of the proposed method, we compared ts results aganst those from the AdaBoost algorthm (commonly used n ths task [10]). Table 3 shows the results from ths comparson. The frst row presents the results of our method, and the second and the thrd rows show the results from AdaBoost usng Naïve Bayes as base classfer and applyng sx teratons. The only dfference from these two runs was the used weghtng scheme. The second row reports the result usng the proposed weghtng scheme, whereas the last row shows the result correspondng to the tf-df weghtng scheme. Table 3. The proposed method aganst the AdaBoost classfer Classfcaton scheme Classfcaton accuracy Rejecton rate Recall of unknown class Our method 95.5% 8.2% 100% AdaBoost (usng our weghtng scheme) 98.9% 29.7% 100% AdaBoost (usng tf-df weghts) 78.2% 6.8% 60% The results from ths experment are very nterestng; they show that our method allowed achevng a better balance between the accuracy and rejecton rates than the tradtonal AdaBoost method. On the other hand, they also show that the proposed weghtng scheme lead to better results than the tf-df weghtng scheme. Moreover, usng the tf-df weghtng scheme t was possble to obtan a small rejecton rate, however, t could not dentfy (and therefore, transfer to a human operator) all ambguous or ncomprehensble utterances. Ths last factor s very relevant snce the msclassfcaton of unclear transcrptons tend to cause a strong dscomfort among users. 5 Conclusons Ths paper proposed a new text classfcaton method that s specally suted to the task of automatc call routng. Ths method dffers from prevous works n two man concerns: the applcaton of a new term weghtng scheme, and the use of a two-step classfcaton approach. Expermental results on a Spansh corpus consstng of 24,638 call utterances showed, on the one hand, that the proposed weghtng scheme s especally approprate for short texts, and on the other hand, that the two-step classfcaton approach allows mantanng an adequate balance between the classfcaton accuracy and the rejecton rate. In other words, these results ndcated that the proposed method s pertnent for the automatc call routng task, whch commonly consders a great number of narrow categores, mbalanced tranng data sets as well as very short text nstances.

9 Another mportant aspect s that the method tself does not rely on language dependent rules, and therefore, t would be easly used n other languages. Fnally, t s mportant to menton that the method can be adjusted (movng the weght thresholds) for each partcular stuaton, n order to reduce the rejecton rate or mprove the classfcaton accuracy. In addton, t s also mportant to comment that the performance of ths method greatly depends on the corpus sze. It requres of a large tran corpora to calculate confdent weghts for the terms. For ths reason, one of the man drectons for future work s the applcaton of sem-supervsed learnng strateges. Acknowledgements: Ths work was done under partal support of CONACYT project grant References 1. Lee, C.-H., Carpenter, B., Chou, W., Chu-Carroll, J., Rechl, W., Saad, A. & Zhou, Q. On Natural Languge Call Routng. Speech Communcaton, Vol. 31, Issue 4, August 2000, pp Carpenter B. & Chu-Carroll, J. Natural Language Call Routng: A Robust, Self-organzng Approach, Proc. ICSLP Sebastan F., Machne learnng n automated text categorzaton, ACM Computng Surveys, 34(1):1 47, Gorn, A.L., Rccard, G. & Wrght, J.H. How may I help you? Speech Communcaton 23, Elsever. 5. Huang Q. & Cox, S.J. Automatc Call Routng wth Multple Language Models. Proc. Workshop on Spoken Language Understandng for Conversatonal Systems, Boston, May Kuo, H.-K., J., Lee, C.-H., Ztoun, I., Fosler-Lusser, E. & Ammcht, E. Dscrmnatve tranng for call classfcaton and routng. In ICSLP-2002, Garfeld S., Wermter S. & Devln S., Spoken Language Classfcaton usng Hybrd Classfer Combnaton. Internatonal Journal of Hybrd Intellgent Systems, Vol. 2, Issue 1, pp 13 33, January Mng Tang, Bryan Pellom, Kadr Hacoglu. Call-Type Classfcaton And Unsupervsed Tranng For The Call Center Doman. Automatc Speech Recognton and Understandng IEEE Workshop, , ISBN ASRU Haffner, P., Tur, G., Wrght, J.H. Optmzng SVMs for complex call classfcaton. Internatonal Conference on Acoustcs, Speech, and Sgnal Processng. ICASSP '03 IEEE. Aprl 2003 Vol. 1, pp , Schapre, R. E. & Snger, Y. BoosTexter: A Boostng-based System for Text Categorzaton. Machne Learnng, 39, pp , Kluwer.