Writer Identification using a Deep Neural Network

Size: px

Start display at page:

Download "Writer Identification using a Deep Neural Network"

Jayson Bridges
5 years ago
Views:

1 Wrter Identfcaton usng a Deep Neural Network Jun Chu and Sargur Srhar Department of Computer Scence and Engneerng Unversty at Buffalo, The State Unversty of New York Buffalo, NY 1469, USA {jchu6, srhar}@buffalo.edu ABSTRACT Most work on automatc wrter dentfcaton reles on handwrtng features defned by humans[6, 4]. These features correspond to basc unts such as letters and words of text. Instead of relyng on human-defned features, we consder here the determnaton of wrtng smlarty usng automatcally determned word-level features learnt by a deep neural network. We generalze the problem of wrter dentfcaton to the defnton of a content-rrelevant handwrtng smlarty. Our method frst takes whether two words were wrtten by the same person as a dscrmnatve label for word-level feature tranng. Then, based on word-level features, we defne wrtng smlarty between passages. Ths smlarty not only shows the dstncton between wrtng styles of dfferent people, but also the development of style of the same person. Performance wth several hdden layers n the neural network are evaluated. The method s appled to determne how a person s wrtng style changes wth tme consderng a chldren s wrtng dataset. The chldren s handwrtng data are annually collected. They were wrtten by chldren of nd, 3rd or 4th grade. Results are gven wth a whole passage (5 words) of wrtng over one-year change. As a comparson, smlar experments on a small amount of data usng conventonal generatve model are also gven. Keywords Deep Neural Network, Wrter Identfcaton, Measure 1. INTRODUCTION It has been beleved that every person has ts consstent wrtng ndvdualty and s dstnct from others. A lot of mportant ssues are related to ths topc, such as sgnature dentfcaton. Therefore, t s very mportant to work out some knd of nvarance n one s wrtng style. A lot of prevous work has acheved great success n ths drecton, such as [6]. Yet most of them are based on human-defned features. In ths paper, we hope to make a step towards automatc Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. Copyrghts for components of ths work owned by others than ACM must be honored. Abstractng wth credt s permtted. To copy otherwse, or republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. Request permssons from Permssons@acm.org. ICVGIP 14, December 14-18, 14, Bangalore, Inda Copyrght 14 ACM /14/1$15. handwrtng feature extracton. The chldren s handwrtng data are collected for the study of the development and evoluton of ndvdualty of handwrtng. Samples are from a large number of students who are learnng or have just learned wrtng. The data are collected annually. The collecton s expected to contnue for 11 years so as to follow the students untl hgh school graduaton. The students wrte the same paragraph each year, twce for prnted ones and twce for cursve ones. Each student was asked to copy the followng paragraph: The brown fox went nto the barn where he saw the black dog. After a second, the black dog saw the fox too. The brown fox was fast and quck. The black dog was not fast and he lost the fox. The fox hd n a hole and wated for the black dog to go home. After the black dog went 4 home, the fox was able to go to the hole he called home and saw all the other foxes. The other foxes were glad to see hm and they all asked hm to tell them about hs day. Cursve samples of the scanned handwrtten paragraph are shown n Fgure. The frequently occurrng word and s used as a base for feature extracton. Features are collected by human examners and nput through a Truthng Tool. For each cursve sample and, 1 features are collected as n Fgure 1. The related entres of the features are used to descrbe chldren s handwrtng style and the dataset s to be used by machne learnng algorthms. Ths knd of handcraft feature extracton has a lot of drawbacks. Frst, because the judgments of shapes and form are made by dfferent human examners. They are very subjectve and unstable. In fact, we fnd numerous conflcts and nconsstency n the delvered feature data. The mputaton of mssng and nconsstent data become a bg problem[1]. Second, t s costly to extract data n ths way, and actually, only a lttle porton of the mage data are extracted and labelled. Features solely from and are also qute nadequate. We need features about other letters and other words to portray a person s ndvdualty. Thrd, these features are based on experence of teachers. However, they are not necessarly ntrnsc. Sometmes we need more subtle and accurate features whch could be hard to descrbe n a natural language. One approach to solvng ths problem s to extract features from strokes and shapes automatcally usng human desgned feature extractors. There are a lot of studes on how to extract useful features (such as usng geometrc propertes of strokes) for wrter verfcaton. Yet, the purpose of

our method s dfferent from some wrter dentfcaton problems. Judcal offcals may want very robust algorthm to tell the dentty of the wrter even f the wrter twsts and tres to deceve hs or her true dentty.

2 our method s dfferent from some wrter dentfcaton problems. Judcal offcals may want very robust algorthm to tell the dentty of the wrter even f the wrter twsts and tres to deceve hs or her true dentty. The study of chldren s handwrtng doesn t requre the ablty of wthstandng ths. Here we would lke to study the wrtng habts and ther way of changng. Thus, all elements related to the wrtng style of chldren are taken nto consderaton except only for the contents. Here we suppose the students were wrtng naturally wthout twstng. Now, n ths paper, we try to fnd out a way to extract nformaton about chldren s cursve handwrtng drectly wthout the need of human examners and also to fnd the sutable features tself. In comparson, our method s about usng dscrmnatve way to fnd out features usng deep neural network (DNN) automatcally wthout the requrement of any pror knowledge. Ths method has several advantages. Frst, we can handle and extract features from numerous data automatcally wthout efforts n feature extractor desgn. It needs preprocessng such lke rule lne removal, cleanng, or denosng but doesn t requre very accurate ones. Second, our method s robust to nose. Because the wrtng ablty of students s lmted, usually there are numerous correctons and marks. These correctons and marks are hard to clean. Thrd, snce the method s not content senstve, t doesn t requre very accurate word or letter segmentaton, whch s very challengng for cursve texts[3]. Fgure 1: The 1 features extracted manually from cursvely wrtten and.. WORD EXTRACTION Chldren wrte ther text accordng to a form shown n Fgure. Each page s of hgh resoluton and contans a lot of nformaton. But they are also very nosy, full of correctons and rrelevant marks and hard to segment. In order to compare the wrtng style between two students, we need to frst reduce the data to an acceptable scale. Thus, we frst extract words from each of the passage and compare wrtng styles on ths level. Our algorthm s very smple but s fast and works on all knds of text mages even though some of them are really wrtten lke a mess..1 Rule lne removal The spatal relatonshps between letters and rule lnes are sometmes taken advantage of. They show one aspect of a person s wrtng habt and personalty. However, n order to smplfy the word segmentaton and sgnfy the mportance Fgure : Examples of scanned paragraphs.

of letter shapes, we need to frst remove rule lnes. There are many complcated methods for underlne removal[]. We have hundreds of mages to be processed, so we want to do t fast.

For the broken lnes, snce they usually appear to be small connected components, they are easy to remove by settng a threshold of the sze of a connected component.

3 of letter shapes, we need to frst remove rule lnes. There are many complcated methods for underlne removal[]. We have hundreds of mages to be processed, so we want to do t fast. Here, because most of the mages scanned are clean, we apply a very quck and smple whle effcent algorthm. For the broken lnes, snce they usually appear to be small connected components, they are easy to remove by settng a threshold of the sze of a connected component. Now the sold lnes have a lot of ntersectons wth the letter parts, we hope that we can remove the sold lnes whle keepng the ntersecton parts because they contan so much crtcal nformaton. Suppose all the rule lnes are perfectly horzontal and unform (n ntensty), an easy way to remove sold lnes s to frst compute the mean value of each horzontal pxel lne and deduct t from each pxel lne n the orgnal mage. However, ths wll also remove the jont parts wth letters. So we must take advantage of the perpheral nformaton around the rule lne to patch back the jont parts. As for mplementaton, before horzontal scannng, we buld a mask by compressng the mage n vertcal drecton to ntensfy the jont part where a stroke comes across the rule lne. And then stretch t back to the orgnal scale by enlargng pxels. The compresson rate could be adjusted. We use ths mask to patch back the ntersecton parts. In experments, we can see, the bgger a stroke s ntersecton angle wth the rule lne, the better the jont part s kept. Snce all the mages are scanned wth lttle devaton angle n drecton, rule lnes are not perfectly horzontal. Fortunately, almost all the mages scanned are n good poston so that even some of the rule lnes are not perfectly horzontal, they just have a very tny slope. Therefore, an easy way to solve ths problem s to separate the whole passage mage vertcally nto a few bandng sub-mages. And n each sub mage, rule lnes are approxmately vertcal. Thus, we smply do all the process each tme n one sub-mage.. Word segmentaton Word segmentaton s a complex task for handwrtng materals lke cursve wrtten texts[7, 3]. However, n ths artcle, the contents of the words are rrelevant. Thus, a very accurate segmentaton s unnecessary. Our method s smply based on the lengths of the word and ther gaps. In the hope of reducng the effect of varous lengths of word samples, we use a normal dstrbuton to exclude outlers and only keep those words wth reasonable lengths. A lot of ms-segmentaton would be deleted n ths process. However, there are stll some ms-clusterng n the data. But snce all we care about s wrtng style nstead of the content. We can just consder them as a normal word. Yet, the kept mages stll have dfferent szes and scales, so we add whte margns to those short ones so as to unfy ther szes. And fnally we resze the mage to 4 8 so that the resoluton s low enough for the DNN to effcently process. Fgure 3: The extracted words from the two paragraphs n Fgure respectvely. 3. DATA PREPARATION We concatenate two word mages together as a data pont and nput t to the lowest layer of the DNN. There are two knds of labels: the one from the same wrter and the one from dstnct wrters. We separate the students consdered nto two groups: the frst group for tranng and the second for testng. The two groups have no ntersectons. For the tranng data, half of them are concatenatons of words from the same wrter and half from dstnct ones. Notce that t s not necessary that the two words are the same. Actually, we randomly choose the combnaton of words from the dataset. For example, f we can extract 5 words from a passage wrtten by a student, we have up to C 5 possble dfferent combnatons of words for the frst half of the data for hs part. See Fgure 4 as an example. The test data have the same dstrbuton but wrtten by totally a dfferent group of students. Fgure 4: A subset of the tranng data. Each of the data s a concatenaton of two words ether wrtten by the same person or not.

4 4. DNN MODEL One advantage of the classfcaton problem s that we can produce numerous ndependent tranng data wth labels to overcome the drawbacks of DNN. Snce tellng the wrtng style of a word needs very hgh level abstracton, the exstng unsupervsed learnng[5] can hardly fnd varatons wth respect to wrtng style whle gnorng the tremendous dfference of shape of dfferent words.e. to fgure out that two dfferent words are wrtten by the same person. Experments show that supervsed learnng s the best choce, so we desgn a DNN model wth 4 hdden layers. See Fgure 5. The valdaton of ths structure s gven n secton 6. The lowest nput layer corresponds to the raw mage nput: an vectorzed mage wth two extracted words concatenated together n t. The nput layer s then connected to upper stacked hdden layers. The adjacent stacked layers l and l + 1 are densely connected by weghts W (l), on top of whch s the output layer wth nodes n t. Between the stacked layers, 1 we use the sgmod functon f (z) = as the actvaton functon so as to ncrease the nonlnearty of our model. 1+exp( z) Therefore, the feedforward operaton could be descrbed as where b s the bas term. z (l+1) = W (l) a (l) + b (l), ( a (l+1) = f z (l+1)). 5.1 Word level feature tranng We use standard backpropagaton to mplement mn-batch stochastc gradent descent[8]. Suppose the loss functon of one data pont of the model s E (W, b; x, y) where W and b are the model parameters, x s the nput and y s the label. We add up the loss functons for one data pont together wth the weght decay regularzaton term as the loss functon of our model. Suppose there are N tranng data. L (W, b) = 1 N E (W, b; x (), y ()) N =1 + λ ( ) w (l) j. l Then we use stochastc gradent descent algorthm to evaluate the parameters. Usng chan rule, we have the partal dervatve of the loss functon wth respect to the value of each node δ (l) = j E (W, b; x, y), for l < n l, where n l s the z (n l) number of layers n the DNN. ( We have ) δ (l) = f z (l) j w (l) j δ(l+1) j. Usng these, we can computer the partal dervatves wth respect to the parameters E (W, b; x, y) w (l) j E (W, b; x, y) b (l) = a (l) j δ(l+1), = δ (l+1). Here we use squared error as the loss functon. 5. Measurement of We can use the DNN descrbed above to defne and compute the smlarty between two wrtten paragraphs. Suppose we use φ (a, b) to denote whether word mage a and word mage b are wrtten by the same person gven by the DNN model we descrbed. If so, φ (a, b) =1, otherwse φ (a, b) =. Suppose we can extract m words from paragraph A and n words from paragraph B. We defne ther smlarty as s (A, B) = 1 m n m n =1 j=1 φ (a, b). Ths knd of measurement could be very useful for the study of chldren s handwrtng ndvdualty development. See Fgure 6 as an llustraton. Fgure 5: Structure of the DNN model 5. TRAINING Fgure 6: Illustraton of how smlarty s computed based on word level features.

Scale of tranng data Average Error Rate 5 pars of words from each student 38% 1 pars of words from each student 33% 3 pars of words from each student 3% Table 1: The relatonshp between the scale of

5 Scale of tranng data Average Error Rate 5 pars of words from each student 38% 1 pars of words from each student 33% 3 pars of words from each student 3% Table 1: The relatonshp between the scale of the tranng data and performance of the algorthm. The error rate s accurate to unt dgt. 6. EXPERIMENTS 6.1 Word level feature tranng We collect 3 students frst page of ther wrtng for tranng and another 3 students for testng. We can extract about 5 words on average from each of the passage. For the tranng set, for each person, we randomly sample 3 pars of words labelled as 1 ; then we randomly sample a par of students 3 3 tmes, each tme samplng a par of words wrtten by them respectvely, labelled as 1. Therefore, we have the same amount of data labelled as 1 or 1. For the testng set, smlarly, for each person, we randomly sample pars of words labelled as 1 ; then we randomly sample a par of students 3 tmes, each tme samplng a par of words wrtten by them respectvely, labelled as 1. We fnd a can reduce the error rate to less than 3%. Ths s not a bad performance snce all the words tested have never been seen by the model and we dd not gve our model any human desgned features. We can conclude that the model has the ablty of capturng wrtng style between dfferent people whle gnorng the very content of the text. The vsualzaton of the weghts n the frst layer s shown n Fgure 7. The amount of data fed nto the DNN s crucal to the performance. The more combnatons we nput, the less overfttng we wll see as shown n Table 1. Durng the tranng the learnng rate α s set to.1. The sparse penalty λ s set to.1. We use a mn-batch stochastc gradent descent backpropagaton to tran the neural network. Fgure 7: Vsualzaton of the weghts of the frst hdden layer. 6. Relatonshp between the structure of the network and performance A lot of experments show that 1-node layers usually gve good performance. Now we evaluate why we choose the network wth ths depth. We compare the performance of the dfferent structures wth the concatenated mage as the nput layer and -node layers as the output layer wth several 1-node hdden layers between them. See fgure 8 showng the relatonshp between depth and performance. Each structure s evaluated 5 tmes. Ths s why we choose structure as our model. Msclassfcaton rate on test data The relatonshp between the depth of NN and Performance Number of mddle 1 node layers Fgure 8: Relatonshp between the depth of the DNN and performance 6.3 Applyng the smlarty on paragraph level Usng the smlarty defned n secton 6, we can see what we get on the paragraph level test data. Frst, we compare the smlarty outcome over data from the same year, then we compare the outcome over one year change. We take another 3 passages wrtten by dfferent students from the frst year data as the test data. We gve two hstograms showng the smlarty appled to passage tself and between passages wrtten by dfferent students. For over one year change comparson, we pck out 5 passages from the frst year data and 5 passages from the second year data and compare the smlarty between them. See fgure 9. Even wth one year gap, passages wrtten by the same students stll show hgher smlarty. 7. CEDAR-FOX RESULTS CEDAR-FOX s a software system for handwrtng comparson developed by the Center of Excellence for Document Analyss and Recognton at Unversty at Buffalo. The system has nterfaces to scan handwrtten document mages. When two handwrtten document mages, one known and another unknown, are presented to the software, t extracts a set of macro, mcro and style features whch are desgned by human bengs[9]. After that t computes a quantty known as Log Lkelhood Rato (LLR) wth these features to descrbe the smlarty between the two document mages. When computng, t uses a generatve model where the Lkelhood Rato s approxmated usng dstrbutons of dstances. A postve LLR value ndcates that the system beleves that the two documents were wrtten by the same wrter. Whle a negatve LLR value ndcates that the sys-

6 LLR Fgure 9: Upper two: smlarty dstrbuton from frst year data. The frst one corresponds to comparson of data from the same passage. The second one corresponds to comparson of data from dfferent students. Lower two: smlarty dstrbuton over one year change. The frst one corresponds to comparson of data from same students but wth one year gap. The second one corresponds to comparson of data from dfferent students and wth one year gap. Fgure 1: CEDAR-FOX results on small amount of data over one year change tem beleves that the two documents were wrtten by dfferent wrters. We pck up a few student samples from the entre dataset and compute ther LLRs over one year change respectvely. We get the results shown n Fgure 1. The system gves good results for students showng postve LLR values but poor results for students showng negatve LLR. It seems the hand-crafted features fal to perform deally for ths problem. CEDAR-FOX s a practcal tool for handwrtng comparson whose effectveness s justfed by many practcal problems. Ths shows the necessty of developng more sophstcated deep learnng methods to crack the problem of chldren s handwrtng. A future generaton of handwrtng software should be able to generate more useful features automatcally. 8. CONCLUSIONS For our method of tranng DNN, the more tranng examples we generate, the better result we get. The major challenge of our method s overfttng. Due to the abstractness of wrtng style, a lot of attrbutes are more sgnfcant than wrtng style, such as the contents, the length of the word and so on. So we need bg data to enhance the ablty of generalzaton. Wth the defnton of smlarty, through smple combnaton we can produce numerous ndependent tranng data to enhance the performance. Why unsupervsed methods don t work? Snce tellng the wrtng style of a word needs very hgh level abstracton, the exstng unsupervsed learnng can hardly fnd varatons wth respect to wrtng style whle gnorng the tremendous dfference of the shapes of dfferent words.e. to fgure out whether two dfferent words are wrtten by the same person. Experments show that supervsed learnng s the best choce. The use of the measurement: The measurement proposed here has varous applcatons. For example, t can help us understand the development of wrtng ndvdualty or study the effectveness of teachers nterventon. 9. REFERENCES

7 [1] Z. Xu and S. N. Srhar. Mssng Value Imputaton: Wth Applcaton to Handwrtng Data. 14. [] W. AbdAlmageed, J. Kumar, and D Doermann. Page rule-lne removal usng lnear subspaces n monochromatc handwrtten arabc documents. Document Analyss and Recognton, pages 768âĂŞ77, 9. [3] G. Loulouds, B. Gatos, I. Pratkaks, C. Halatss1. Lne and word segmentaton of handwrtten documents. 1st Internatonal Conference on Fronters n Handwrtng Recognton, pages 47-5, 8. [4] M. Bulacu and L. Schomaker. Text-ndependent wrter dentfcaton and verfcaton usng textural and allographc features. Pattern Analyss and Machne Intellgence, IEEE Transactons, 9(4):71-717, 7. [5] G. E. Hnton, and R. R. Salakhutdnov. Reducng the dmensonalty of data wth neural networks. Scence 313(5786):54-57, 6. [6] S. N. Srhar., S. H. Cha, H. Arora and S. Lee. Indvdualty of handwrtng. Journal of Forensc Scences, 47(4):856-87, [7] J. Park, V. Govndaraju, S. N. Srhar. Effcent word segmentaton drven by unconstraned handwrtten phrase recognton. Document Analyss and Recognton, [8] R. Hecht-Nelsen.Theory of the backpropagaton neural network. Neural Networks, [9] S. N. Srhar, C. Huang, H. Srnvasan. On the Dscrmnablty of the Handwrtng of Twns. Journal of Forensc Scences, 53():43-446, 8

Lecture 5: Multilayer Perceptrons

Lecture 5: Multilayer Perceptrons Lecture 5: Multlayer Perceptrons Roger Grosse 1 Introducton So far, we ve only talked about lnear models: lnear regresson and lnear bnary classfers. We noted that there are functons that can t be represented