Handwritten English Character Recognition Using Logistic Regression and Neural Network

Transcription

1 Handwrtten Englsh Character Recognton Usng Logstc Regresson and Neural Network Tapan Kuar Hazra 1, Rajdeep Sarkar 2, Ankt Kuar 3 1 Departent of Inforaton Technology, Insttute of Engneerng and Manageent, Salt Lake, Kolkata Departent of Inforaton Technology, Insttute of Engneerng and Manageent, Salt Lake, Kolkata Departent of Inforaton Technology, Insttute of Engneerng and Manageent, Salt Lake, Kolkata Abstract: Hand wrtten character recognton s a challengng task often resultng n abguous labels. Usng the concepts of Machne learnng we have tred to develop an Optcal Character Recognton (OCR) syste where an algorth s traned on a data set of known letters and then can learn to accurately classfy new data. Our optcal character recognton (OCR) syste for handwrtten Englsh characters coprses of two steps-generatng tranng set data usng an OCR tool and then Applyng dfferent achne learnng algorth on the tranng set and starts the learnng process. A varety of algorths have shown good accuracy for the handwrtten letters, two of whch are looked here. Keywords: neural network, classfcaton, optcal character recognton, regularzaton, logstc regresson 1. Introducton Optcal Character Recognton (OCR) an area of coputer scence that started developng as early as 1950, currently encopasses two prevously dstnct areas - pure optcal character recognton, usng optcal technques such as rrors and lenses and dgtal character recognton, usng scanners and coputer algorths. The ntent of ths paper s to focus on recognton of sngle handwrtten character, by vewng t as a data classfcaton proble. Analyss of the result should gve us an dea on the vablty of the algorths pleented for use n coplete OCR applcaton. Success of optcal character recognton depends on a nuber of factors, two of whch are feature extracton and classfcaton algorths. We fnd Englsh OCR an nterestng proble. Ths s anly because OCR tself les n between one densonal natural language process (NLP) and three densonal real world projected down to age n coputer vson. The way we extract features for tranng/testng our odel s by frst wrtng the text on a whte or lght background. Then we use the detecton of whte spaces as a eans of segentng the age nto regons that contans lnes of text. We can reapply the process to segent the lne regons nto saller regons that contan words and letters. Whle soe OCR algorths work wth grey-scale ages, any convert ther nput nto bnary ages durng the early stages of processng. For our pleentaton we have done both, but have used the grey-scale ntensty values as nput features. We have not used bnarzed age because t outputs ntensty n ters of only 0 and 1 whch ay not gve as good result as a grey-scale age. Gven that we have age regons that contan text, whether sngle word regons or whole slabs of text, the goal here s to dentfy age pxels that belong to text and those that belong to the background. In our project we have used a tool [4] for feature extracton whch s the ntensty of each pxel. Ths tool s copletely open source, but we have ade certan odfcatons to t accordng to our need. The data generated by ths tool s then used to tran our algorth. In our work, ost characters are well structured and well-separated [2], whch s dfferent fro general vson recognton tasks where objects have uch varaton. 2. Related Work 2.1 OCR tool Optcal character recognton s a cobnaton of natural language processng (NLP) and Coputer Vson. In our applcaton, language odelng s less portant snce we have ore clues fro the age lkelhood, and the vson clues are ore robust snce t s an ntrnsc 2Dage. Below s gven how our OCR tool works: 1. In frst phase, the age contanng the text s converted nto gray-scale (preprocessng). Fgure 1: gray-scaled then bnarzed age 2. In next phase, the syste fnds characters by layout analyss. Here the text s separated nto unconnected characters. Paper ID: NOV

2 3. The tool then generates a labeled data set for the dentfed characters. The data generated s used to tran our odel. 4. Lastly test characters are classfed based on odel fro tranng data. Ths s referred to as recognton. 2.2 Classfcaton In achne learnng, classfcaton [1] s the proble of dentfyng to whch of a set of categores a new observaton belongs, on the bass of a tranng set of data contanng observatons whose category ebershp s known. Classfcaton s an exaple of pattern recognton. 3. Data Sets Snce the algorths we used are supervsed learnng algorths so we had to develop a way to generate the labeled data sets. For ths we have used our OCR tool whch gves us our requred labeled data set as shown n the followng fgure: In the ternology of achne learnng, classfcaton s consdered an nstance of supervsed learnng,.e. learnng where a tranng set of correctly dentfed observatons s avalable. The correspondng unsupervsed procedure s known as clusterng, and nvolves groupng data nto categores based on soe easure of nherent slarty or dstance Defnton We frst revew the general goal of Machne Learnng Algorth: A coputer progra s sad to learn fro experence E wth respect to soe class of tasks T and perforance easure P, f ts perforance at tasks n T, as easured by P, proves wth experence E. For classfcaton proble, the task s to tell thngs dfferent apart and the experence E s to learn the sutable representaton of the true data dstrbuton based on the perforance easured by P on tranng data. It can be vewed as an optzaton proble over a hypothess space h. In the paraetrc settng, the objectve functon s gven by ƒ (;x) where s the odel paraeter and x s a gven saple. An eprcal objectve s gven n equaton (1): ƒ () = 1 =1 ƒ ( ; x ) (1) Fgure 2: OCR tool Ths tool recognzes ndvdual characters and for each character outputs a 20*20 atrx. Ths gves us a feature set of 400 features for each tranng exaple. To obtan these features we frst convert our age to gray scale. Then each ndvdual character s recognzed and the atrx s fored by the ntensty value of every pxel. 4. Algorths Ipleented We realzed that the essence of the project was to understand the ntrcaces of dfferent achne learnng algorths and to learn whch algorth gves good result for whch use case. Wth ths phlosophy n nd, we wrote our own pleentaton of Regularzed Logstc Regresson [3] and Neural Network. The followng fgure shows how a supervsed learnng algorth works: here x 1, x 2,,x are our saples Paraetrc vs Nonparaetrc In ters of odelng the hypothess space, classfcaton algorths can be dvded nto paraetrc ones and nonparaetrc ones. In the paraetrc settng where the hypothess space s a faly of paraeterzed functon, no atter Frequentst, Bayesanst or Learnng Theorst, no atter Dscrtve, Generatve or Algortc, we end up wth an object functon whch we have to optze Bnary-class vs. Mult-class In practce Neural Networks can be easly appled for ult-class classfcaton. But, snce Logstc Regresson s a bnary classfer, there s no drect ethodology on ult-class probles. One natural approach s to reduce the ult-class proble to several bnary ones. Ths can be done ether by one-vs.-one or one-vs.-all schees. Fgure 3: Supervsed learnng algorth workng 4.1 Regularzed Logstc Regresson Logstc regresson s a bnary classfer. But our project had 26 dfferent classes, so we pleented a one vs. all extenson to classfy our 26 characters. The goal of ths algorth s to nze the cost functon gven by equaton (2): Paper ID: NOV

3 J () = 1 { y log h =1 x + (1 y ) log (1 h x ) } (2) Where, h (x) = 1 logstc functon. (1+e TX ) s the sgod functon or Overfttng wth Logstc Regresson If we have large nuber of features, the learned hypothess ay ft the tranng set well, but fals to generate new exaples,.e. predct the output on new data. Ths s called over fttng. Snce our data set has large nuber of features (400), there s a hgh probablty that our hypothess wll overft our tranng data. To overcoe ths there are anly two ways: Reducng the nuber of features Fgure 4: Logstc functon Interpretng hypothess output When our hypothess (h (x)) outputs a nuber, we treat that value as the estated probablty that y=1 on nput x. For exaple, If x s a feature vector wth x 0 =1 and x 1 = tuor Sze and we get h (x) = 0.7, ths tells that the patent has a 70% chance of the tuor beng algnant. We can wrte ths wth the followng notaton: h (x) = P (y=1 x ; θ) (3) Equaton (3) represents the probablty that y=1, gven x, paraeterzed by θ. To reduce the nuber of features we can ether select whch features to keep or we can use Model Selecton algorths. But, n reducng the nuber of features we lose soe nforaton. We can also select those features whch nze data loss, but even so, soe nforaton s stll lost Regularzaton The second and uch better way to resolve overfttng s by usng a technque whch s called regularzaton. Ths allows us to keep all features, but reduces the agntude of paraeters. The regularzed logstc regresson cost functon s gven n equaton (4): J () = 1 { y log h =1 x + (1 y ) log (1 h x n 2 ) } + λ/2 j Where, λ s the regularzaton paraeter, x () s the set of features, y () s the set of labels, s the paraeter upon whch the optzaton s done, n s the nuber of features for each tranng exaple and s the nuber of tranng exaple. 4.2 Neural Network j =1 (4) In artfcal Neural Network, a neuron s a logstc unt. We feed nput va nput wres, the logstc unt does the coputaton and the output wre sends the output. The logstc coputaton s the sae Logstc regresson hypothess calculaton. Fgure 6: A neural network In the above fgure x 1, x 2, and x 3 consttute the nput layer, a , a 2 and a 3 consttute the hdden layer and h (x) whch produces the output s called the output layer. We can have as any hdden layer as our requreent Types of classfcaton proble wth Neural Network There are bascally two types of classfcaton: Fgure 5: A sngle neuron. Bnary classfcaton The above fgure s an artfcal neuron wth a sgod (logstc) actvaton functon. The vector s also called the weghts of the odel. For bnary class, there s one output, ether 0 or 1. So there s only one output node and the value s gong to be a real nuber and s L = 1. So the output vector y s a k- densonal vector of real nubers. Paper ID: NOV

4 . Mult-class classfcaton Internatonal Journal of Scence and Research (IJSR) For ult-class there are k dstnct classfcatons. Typcally k s greater than or equal to 3. Here s L = k Ipleentaton detals For Neural Network we have pleented the backpropagaton algorth. Our Neural Network has one nput layer, one hdden layer, and one output layer. The output layer had 26 neurons or nodes each for one class. The cost functon that was nzed n ths algorth s gven n equaton (5): J () = 1 =1 K k=1 y k log (h x ) k + (1 y k ) log (1 ( h x ) k + λ 2 L 1 s s l 2 l l+1 l=1 =1 j =1 j (5) Where K = nuber of dstnct classes, s l = k. Whle pleentng the hdden layer we tred wth any dfferent nubers of neurons and fnally found that the nuber of nodes n the hdden layer should be around ean of nput layer and output layer. There are bascally two halves to the Neural Network cost functon represented by equaton (5) Frst half 1 K k=1 y k log (h x =1 ) k + (1 y k ) log (1 ( h x ) k (5a) Ths says, for each tranng data exaple (.e. 1 to the frst suaton) su for each poston n the output vector Second half λ 2 L 1 s l =1 s l+1 j=1 j l 2 l=1 (5b) Ths assve regularzaton suaton ter s also called a weght decay ter. The regularzaton ter s very slar to that n Logstc Regresson cost functon gven n equaton (4). The fgure gven below shows the nzaton of the cost functon: Ths ay ean t looks lke J () s decreasng, but n realty t ay not be decreasng by as uch as t should. So usng a nuerc ethod to check the gradent can help dagnose a bug. Gradent checkng helps ake sure an pleentaton s workng correctly. 5. Result We perfor our tranng by takng saples of handwrtng fro two dfferent ndvduals. We tran our classfers for both lower and upper case letters. Our tranng data contans 26 characters, and totally about 1100 tranng saples for each ndvdual and each case. Testng data contans about 100 characters. The followng tables show tranng versus testng accuracy for the two dfferent learnng algorths we pleented: Table 1: Frst person, lower case Tranng Algorth Testng Accuracy Accuracy Logstc Regresson 100% 79.22% Neural Network 82.64% 59.74% Table 2: Second person, lower case Algorth Tranng Accuracy Testng Accuracy Fgure 7: Cost functon nzaton Paraeters for controllng appng fro one layer to next If a neural network has s j unts (neurons) n layer j and s j+1 unts n layer j+1 then j wll be of densons [s j+1 * s j +1] Gradent checkng Logstc Regresson 100% 74.36% Neural Network 90.09% 55.13% Table 3: Frst person, upper case Algorth Tranng Accuracy Testng Accuracy Logstc Regresson 100% 88.46% Neural Network 97.52% 79.49% Backpropagaton has a lot of detals; sall bugs can be present and run t. Paper ID: NOV

5 Table 4: Second person, upper case Algorth Tranng Accuracy Testng Accuracy Logstc Regresson 100% 78.21% Neural Network 81.79% 50.00% 6. Concluson Though our proposed ethod recognzes handwrtten characters wth about 80% accuracy for Logstc Regresson and about 60% for Neural Networks, t s stll not good enough for practcal use. Constrant to the dffculty of obtanng ground truth data, we here only conduct experent on a relatvely sall dataset. In future work, we wsh we can tran these ethods wth ore data and have further testng. 7. Future Scope 1. We used raw pxel ntensty as out features. But we can use dfferent ethod to extract features whch ght prove our perforance. 2. In Neural Network we used only one hdden layer. Neural Networks n coputatonally very expensve and t takes very long te to tran n our achnes. So, we can try to run Neural Networks wth ore nuber of hdden layers on hgh perforance coputng achnes. 3. We can develop an Androd/OS applcaton that dentfes the handwrtng n real te on clckng a pcture of t. References [1] Bhandar, Adtya, Aeya Josh, and Roht Patk. "Brd Speces Identfcaton fro an Iage." [2] URL: handar,%20aeya%20josh,%20roht%20patk,%2 0Brd%20Speces%20Identfcaton%20fro%20an% 20Iage.pdf [3] Shh, Ychang, and Dongla We. "Machne Learnng Fnal Project: Handwrtten Sanskrt Recognton usng a Mult-class SVM wth K-NN Gudance." URL: = [4] George Marguls- Optcal Character Recognton: Classfcaton of handwrtten Dgts and Coputer Fonts. URL: OptcalCharacterRecognton.pdf [5] l-ocr-tool. URL: Author Profle Tapan Kuar Hazra copleted hs M.E degree fro Jadavpur Unversty, Kolkata, West Bengal, Inda. Snce fro 2003, he s workng as Assstant Professor of Departent of Inforaton Technology at Insttute of Engneerng & Manageent, Salt Lake, Kolkata, West Bengal, Inda. Hs research nterest ncludes Desgn and Analyss of Algorths, Iage Processng, Natural Language Processng, Sentent Analyss, Machne learnng, Cryptography. Rajdeep Sarkar, s pursung hs fnal seester B.Tech degree n Inforaton Technology fro Insttute of Engneerng and Manageent afflated to Maulana Abul Kala Azad Unversty of Technology, West Bengal. Hs research nterest ncludes Machne learnng and age processng. Ankt Kuar s pursung hs fnal seester B.Tech degree n Inforaton Technology fro Insttute of Engneerng and Manageent afflated to Maulana Abul Kala Azad Unversty of Technology, West Bengal. Hs research nterest ncludes Machne learnng. Paper ID: NOV