On Computing Strength of Evidence for Writer Verification

Transcription

1 On Computng Strength of Evdence for Wrter Verfcaton Shrvardhan Kabra Harsh Srnvasan Chen Huang Sargur Srhar Center of Excellence for Document Analyss and Recognton (CEDAR) Department of Computer Scence and Engneerng, Unversty at Buffalo {skabra, hs32, chuang5, Abstract The problem of Wrter Verfcaton s to make a decson of whether or not two handwrtten documents are wrtten by the same person. Provdng a strength of evdence for any such decson s an ntegral part of the wrter verfcaton problem. The strength of evdence should ncorporate () The amount of nformaton compared n each of the two documents (lne/half page/full page etc.), () The nature of content present n the document (same/dfferent content), () Features used for comparson and (v) The error rate of the model used for makng the decson. Ths paper descrbes the statstcal model used for wrter verfcaton and also ntroduces a mathematcal formulaton to nclude the above four mentoned parameters, for calculatng strength of evdence of same/dfferent wrter. The statstcal model uses Gamma and Gaussan denstes to parametrcally model the dstance space dstrbuton arsng from comparng ensemble of pars of documents. The strength of evdence s mapped to a 9-pont qualtatve scale for the decson; one that s often used by questoned document examners. Experments and results show that wth ncrease n nformaton content from just a sngle word to a full page of document, the verfcaton accuracy of the model ncreases. of Kullback-Lebler(KL) dvergence) to one of the dstrbuton(same), t s about the same for the other dstrbuton(dfferent). Frstly ths paper shows that, modelng the same dstrbuton wth Gamma and the dfferent dstrbuton wth Gaussan s better. Secondly, ths paper focuses on a method to provde a strength of evdence of same/dfferent that ncorporates () The amount of nformaton compared n each of the two documents(lne/half page/full page etc.), () The nature of content present n the document (same/dfferent content), () Features used for comparson and (v) The error rate of the model used for makng the decson. The frst two of the parameters above were not ncluded n framework of strength of evdence n []. The strength of evdence s then calbrated nto a standard nne-pont scale used by questoned document examners. 2 Wrter Verfcaton Model The three major components of the verfcaton model nclude () Identfyng dscrmnatng elements () Mappng from feature to dstance space by usng smlarty measure and ()Parametrc modelng of the dstance space dstrbuton. Each of the three s brefly descrbed below. 2. Features - Dscrmnatng elements Introducton Wrter verfcaton s the task of determnng whether two handwrtng samples were wrtten by the same or by dfferent wrters, a task of mportance n Questoned Document Examnaton (QDE). A statstcal model for ths task was descrbed n []. The model uses two dstance space dstrbutons resultng from comparng features from ensemble of pars of documents wrtten by () same wrter and () dfferent wrters. It was shown that a Gamma densty s a better ft than Gaussan denstes for modelng these dstrbutons. Although a Gamma s a better ft(n terms Features for wrter verfcaton have been splt nto Macro (global) and Mcro (Local) features n [2] and features are termed as dscrmnatng elements (DE [3]). The statstcal model to be descrbed can be used wth any other set of features as well. Macro features(3 of them ) are gray-scale based (entropy, threshold, no. of black pxels), contour based (external and nternal contours), slope-based (horzontal, postve, vertcal and negatve), stroke-wdth, slant and heght. These are real valued features. Mcro features are for characters, b-grams(two characters) and words. A set of 52 bnary-valued mcro-features correspondng to gradent (92 bts), structural (92 bts), and

2 concavty (28 bts) whch respectvely capture the fnest varatons n the contour, ntermedate stroke nformaton and larger concavtes and enclosed regons, e.g., [4] are used for characters. For b-grams and words, 24 such bnary valued GSC features [5] are used. The number of local features are dependent on the recognton of characters n the document. To assst recognton and ncrease the amount of local features, lexcon-based word recognton and transcrpt mappng approaches [6] can be used. The number of local features also depends on the amount of nformaton n the document(full page/half page etc.). 2.2 space dstrbuton Once a set of features descrbed above n Secton 2. are avalable for two documents, they can be compared. The comparson results n mappng from feature space to dstance space. The macro features are real valued and hence the mappng to dstance space s just the absolute dfference between the two feature values. On the other hand, the smlarty between bnary valued feature vectors for local features can be calculated usng a number of dfferent measures such as Hammng dstance, Eucldean dstance etc. A detaled descrpton of smlarty measures for bnary features s descrbed n [7]. After much expermentaton, the correlaton smlarty was decded as the best smlarty measure. 2.3 Parametrc model The dstrbuton n dstance space can be modeled usng known probablty densty functons (pdf). Assumng that smlarty data can be acceptably represented by Gaussan or Gamma dstrbutons, pdfs of dstances condtoned upon the same-wrter and dfferent-wrter categores for a sngle feature x have the parametrc forms p s (x) N(µ s,σs), 2 p d (x) N(µ d,σd 2) for the Gaussan case, and p s(x) Gam(a s,b s ), p d (x) Gam(a d,b d ) for the Gamma case. The Gaussan and Gamma densty functons are as follows. Gaussan: p(x) = (2π) /2 σ exp 2 ( x µ σ )2. Gamma: p(x) = x a exp ( x/b) (Γ(a)) b. Estmatng µ and σ from samples usng the a usual maxmum lkelhood estmaton, the parameters of the gamma dstrbuton are calculated as a = µ 2 /σ 2 and b = σ 2 /µ. For macro features, we model both categores by Gamma dstrbuton as p s (x) Gam(α s,β s ) and p d (x) Gam(α d,β d ). For mcro features, whle the same-wrter category s modeled as p s (x) Gam(α s,β s ) for Gamma dstrbuton, the dfferent-wrter s modeled as p d (x) N(µ d,σd 2 ) for Gaussan dstrbuton. As an example, dssmlarty hstograms correspondng to the same wrter and dfferent wrter dstrbutons for slant (macro feature) and for the letter e (mcro feature) are shown n Fgure (a) and (b) respectvely. Condtonal parametrc pdfs for slant and for e are shown n Fgure (c) and (d). Probablty densty (a) (c) Probablty densty x x (b) (d) Fgure. Hstograms and parametrc probablty densty functons Once the dstrbutons are modeled, the learnng phase s complete. A new par of unseen documents when compared results n a N dstance values, for each feature compared. For eg. f there were common characters n the two documents, and bgrams and words common, the value of N would be 3 Macro + Mcro features = 23. It s evdent that the nature of the document affects the number of mcro features but not the macro features. For one dstance value x, {...N}, the Lkelhood rato (LR) s gven as LR(x) = p s (x)/p d (x). Consderng the features as ndependent, we can have the LR for N of them as N LR(x ) = N p s (x )/p d (x ). The log lkelhood rato (LLR) s gven as LLR = N log p s (x ) log p d (x ) 2.4 Dscusson on LLR The understandng of the log lkelhood rato for the problem of wrter verfcaton s the key before the formulaton of strength of evdence s presented. Frstly t s mportant to note that the magntude of the LLR alone cannot provde a strength of evdence, though ths s not obvous drectly. Two reasons for ths are the followng. The reason can be llustrated wth an example. Lets say that a LLR of + was obtaned when only one word was avalable n the two documents ndcatng same wrter par. If the same LLR of + was obtaned when a whole page of document was compared, then ths s a weaker case. Ths s because, f a whole page of document were avalable, one would expect a larger postve LLR. Ths then a relatvely small LLR value n magntude can represent a strong strength of evdence. 2. The doman of LLR s ( nf,nf) and hence t cannot be calbrated onto the requred 9 qualtatve scale (dscussed later n Secton 3. Even wth the use of a

3 smoother lke a sgmod or a hyperbolc tangent, the resultng values le n the extremty and does not take nto account the amount of nformaton compared between the documents, the features used and the nature of the content compared. Hence there requres a new way to map from LLR to a qualtatve assessment of strength of evdence. It s evdent that the value of N (the number of features) drectly affects the magntude of the LLR. Below are two key factors that nfluence the value of N.. The amount of nformaton compared n the two documents (I). When two full page documents are compared, then t s more lkely that more characters are common between them and hence greater s the value of N. If the model s perfect, then each of the terms n the summaton s postve for same wrter pars and negatve for dfferent wrter pars. Hence, the magntude of LLR s hgher when documents wth more nformaton s compared. 2. The nature of content(same/dfferent) n the two documents (C). When two documents beng compared have the same content(text) n them, then t s obvous that more characters/b-grams/words recognzed n each of them wll be common. Hence, ths also ncreases the value of N and results n a hgher magntude of LLR. Also t s mportant to note that, when N s fxed (for eg. when only macro features are used), the range of the LLR values should not vary much. However, the accuracy of the model defned as percentage of correctly made decsons(same/dfferent) s stll nfluenced by the amount of nformaton I and the nature of the content C. For example, when only macro features are used, the feature values are more accurate when more nformaton s present n the document and as a result the LLR values classfy the par as same and dfferent better. 3 Strength of Evdence Havng observed two mportant reasons that affect the LLR as mentoned n Secton 2.4, the formulaton of strength of evdence s parameterzed based on two thngs () The amount of nformaton compared(i) and () Whether the documents beng compared have the same or dfferent content(c). In ths paper, we dscretze the values that I can take on as one of word,lne,multple lnes,half page,full page. Words of dfferent lengths(short,medum and long) and words made of purely numbers were all consdered as belongng to the type word. The two dfferent values C can take on s Same Content,Dfferent Content. The value of I can be automatcally found durng the processng of the handwrtten document whch conssts of lne and word segmentaton. The algorthm used for lne segmentaton s dscussed n [8] and that for word segmentaton s usng a Neural Network to decde whether or not the gap between two connected components s a word gap or not. Hence usng the number of lnes L and the number of words n each lne W {...L}, the value of I s decded as follows: [L =,W = I = word],[l =,W > I = lne],[l > & L < 4 I = multple-lnes],[l 4 & L < 8 I = half-page],[l 8 Ifull-page]. Smlarly for the value of C, f the number of common words between the two documents s greater than 8% of the number of words n the smaller document(n terms of number of lnes), then the value of C s Same Content, or else t s dfferent content. 3. Mathematcal formulaton For each possble pars of settngs for I and C, the dstrbuton of the LLR as observed on a valdaton set of ensemble of pars s obtaned. Ths ensemble of pars conssts of both, pars from same as well as from dfferent wrters. The number of such pars from the same and dfferent wrters are kept the same to avod a bas n the dstrbuton of LLR. Let D c represent the dstrbuton of LLR for I = and C = c. Further, let Dc S represent the subset of D c where the samples truly belonged to the same wrter and let Dc D represent the subset of D c where the samples truly belonged to dfferent wrters. It s clear that D c = Dc S DD c. Here t s mportant to note that the dstrbuton Dc S and DD c wll be dfferent for dfferent sets of features used. For eg., the dstrbuton can be further parameterzed by a thrd varable that measures whch feature set was used (macro only or macro+mcro). We leave the dscusson of ncluson of ths thrd parameter to the experments and results secton. Usng the dstrbutons Dc S and DD c, and for any gven value of LLR L, two percentages can now be obtaned () Pc S: Percentage of samples n Dc S that had LLR values > L and () Pc D: Percentage of samples n DD c that had LLR values > L. To be verbose, Pc S represents the percentage of same wrter cases n the valdaton set that had LLR values even larger than the one for ths. Ths mples that Pc S represents the percentage of same wrter cases n the valdaton set that were stronger than the current case. Smlarly, Pc D represents percentage of dfferent wrter cases that were weaker than the current case. Mathematcally, they are defned as n equaton. P S c = DS c >L D S c P D c = DD c >L D D c ()

4 Scale Opnons for same P S c Identfed as same Hghly probably same Probably same Indcatng same No concluson Table. Rules for obtanng the 9 pont scale(-5) when LLR L s +ve Scale Opnons for dfferent P D c 5 No concluson Indcatng dfferent Probably dfferent Hghly probable dfferent Identfed as dfferent Table 2. Rules for obtanng the 9 pont scale(5-9) when LLR L s -ve where represents cardnalty. It s clear that Pc S = Pc S wll represent that percentage of samples n DS c that had LLR values L and smlarly we defne Pc D = D and Pc represent the complement of P c S and P c D Pc D. P c S respectvely pont scale A standard 9 pont scale s used by questoned document examners to verfy whether two documents are wrtten by the same person or not. Ths scale s as follows: [-Identfed as same, 2-Hghly Probable same, 3-Probably same, 4-Indcatng same, 5-No concluson, 6-Indcatng dfferent, 7-Probably dfferent, 8-Hghly probably dfferent and 9-Identfed as dfferent]. The sgn(+ve,-ve) of the LLR L between a par of documents makes a decson of same or dfferent wrter. The strength of evdence s based on ths decson. The scale 9 for a partcular par of document can be obtaned usng ether Pc S (f L < ). (In both cases, we are evaluatng the percentage of samples that were stronger than the current case.) These two values can be calculated usng Equaton provded and c are known. The begnnng of ths secton descrbed the method to calculate these and c. If the LLR L s +ve, then the opnon scale s n the range -5 and n the range 5-9 f t s -ve. Note that, n ether cases, the scale 5-No concluson can be obtaned. Table and 2 summarzes the rules for obtanng the 9 pont scale for +ve and -ve LLR values. 4 Experments (f L > ) or P D c All experments were conducted usng the CEDAR-FOX system [9] that s an applcaton desgned for wrter verfcaton. A total of 824 wrters, wth each wrter havng wrtten 3 samples of full page document s the sze of the Informaton I Content C Sys. Accuracy Mn.LLR Max.LLR Lne Same 8.46% Dfferent 62.98% Multple lnes Same 88.7% Dfferent Half page Same 93.56% Dfferent 94.4% Full page Same Table 3. System accuracy and range of LLR when only Macro features are used total learnng/valdaton set to obtan the four dstrbutons Pc S,P c D S D,Pc and Pc. These documents were full page documents, and hence smaller content half-page, lne, word etc. were derved from ths set. Another set of the same sze was used as the testng set. The experments were conducted prmarly to show that the system accuracy and the range of LLR values obtaned do vary wth the amount of nformaton content I and the nature of the content C. The strength of evdence presented above s just a mathematcal formulaton to provde for a qualtatve assessment of the models decson. A thrd parameter other than I and C s ntroduced here as was told n Secton 3.. Ths parameter s ntroduced due to the nteractve nature of the system that has the capablty of nteractvely turnng on/off the mcro features and hence Tppett plots(dstrbutons) nclusve/exclusve of mcro features can also be obtaned. The results are presented for each of settng of ths parameter. 4. Results wth macro features only Wthout the mcro features (only macro avalable), the value of N (number of features) to calculate the LLR s a constant and hence as expected, the dstrbuton of LLR values does not vary much. The mnmum and maxmum values of the LLR are shown n Table 3. A row for word level s not ncluded n the table snce for one word, the macro features are extremely nosy. Ths s because, global document features cannot be represented well wth just one word. But, ths row s ncluded for experments when macro as well as mcro s used (see Secton 4.2). Also the row correspondng to full page dfferent content s also not avalable due to lack of such data. The system accuracy column n the Table 3 s the percentage of tmes the system decded on same/dfferent wrter correctly. Although the range of LLR values s the same, t s evdent from the table that as the nformaton present n the document ncreases, the system accuracy also ncreases and saturates when half page of document s present. The mappng from LLR to 9 pont scale strength of evdence s the same as dscussed n Secton 3.. The LLR dstrbutons correspondng to each row of the table s stored from performng experments on the valdaton set. Once ths s done, equaton, Table and Table 2 are used to calculate the 9 pont scale.

5 No of letters Word/Strng Sys. Accuracy Mn.LLR Max.LLR 9 Allentown 8.43% Street 73.3% Elder 8.9% * York,Loop 74.63% * Apt,New 74.63% % * 829,3 73.5% % Table 4. System accuracy and range of LLR when only one word/strng s used. It s seen that dscrmnablty power of word s more than numerals and longer words gve better accuracy. * denotes that the average accuracy was taken when two words of same length were consdered. Informaton I Content C Sys. Accuracy Mn.LLR Max.LLR Lne Same 86.4% Dfferent 62.98% Multple lnes Same 93.8% Dfferent Half page Same 93.8% Dfferent 94.78% Full page Same 95.75% Table 5. System accuracy and range of LLR when only Macro and Mcro features are used 4.2 Results wth macro and mcro features Wth the ncluson of mcro features, the range of LLR values as well as the system accuracy vares. Table 4 shows how the system accuracy vares wth the number of characters avalable when only one word s avalable n both documents. It s seen that the system accuracy on longer words s better than those n shorter words. Also the dscrmnablty power of words wth characters s seen to be greater than strngs of numbers. Table 5 shows the system accuracy and the LLR range when macro and mcro features are used for other page types. The calculaton of strength of evdence for a new unseen test case s as before. 4.3 Dscusson on strength of evdence From the above experments t s conclusve that the LLR values and hence the model s accuracy n decson makng depends on the amount of nformaton as well as the nature of content. When presented wth a new par of documents to be assessed, the followng nformaton s used to calculate the strength of evdence ()Amount of nformaton I (word,lne,multple-lnes,half-page,full-page), () Nature of content compared C (same content,dfferent content), () Lst of features compared (only Macro or Macro+Mcro). and (v)llr value obtaned by the model L. The frst three of the above decdes on whch LLR dstrbuton(d c as n Secton 3.) to use for the assessment. The LLR value s then used to calculate the percentages and 9 pont scale as n Equaton and Table and 2. Note that the 9 pont scale opnon s condtoned on the above four mentoned quanttes and hence t suffces to say that the strength of evdence s reflectve of the complete model/setup used. 5 Summary and Concluson A statstcal model for wrter verfcaton usng Gamma and Gaussan denstes to model the dstance space dstrbuton was shown as a better ft than earler used models[]. A mathematcal formulaton to provde for a qualtatve 9 pont scale assessment of strength of evdence was ntroduced and t was shown that ths formulaton takes nto account () Amount of nformaton present n the documents, () The nature of the content compared n the two documents and () The set of features used(macro/mcro) for comparson. Experments and results on a large set ndcated that the models accuracy and range of LLR values does ndeed depend on all of the above mentoned factors and hence the correctness of the formulaton of strength of evdence. References [] S. Srhar, M. Beal, K. Band, V. Shah, and P. Krshnamurthy, A statstcal model for wrter verfcaton, August 25, pp. pp [2] S. Lee, S. Cha, and S. N. Srhar, Combnng macro and mcro features for wrter dentfcaton, SPIE, Document Recognton and Retreval IX, San Jose, CA, pp , 22. [3] R. A. Huber and A. M. Headrck, Handwrtng dentfcaton:facts and fundamentals, CRC Press, 999. [4] S. N. Srhar, S. H. Cha, and S. Lee, Indvdualty of handwrtng, n Journal of Forensc Scences, 22, pp [5] B. Zhang, S. N. Srhar, and C. Huang, Word mage retreval usng bnary features, n SPIE, E. H. B. Smth, J. Hu, and J. Allan, Eds., vol. 5296, 24, pp [6] C. Huang and S. Srhar, Mappng transcrpts to handwrtten text, October 26, pp. pp [7] B. Zhang and S. N. Srhar, Bnary vector dssmlarty measures for handwrtng dentfcaton, Proc. Document Recognton and Retreval X, Santa Clara, CA, pp. pp 28 38, January 23. [8] M. Arvazhagan, H. Srnvasan, and S. Srhar, A statstcal approach to handwrtten lne segmentaton, February 27. [9] S. N. Srhar, B. Zhang, C. Toma, S. Lee, Z. Sh, and Y.-C. Shn, A system for handwrtng matchng and recognton. Proc. Symposum on Document Image Understandng Technology, 23, pp