Font Recognition in Natural Images via Transfer Learning

Transcription

1 Font Recognton n Natural Images va Transfer Learnng Yzh Wang, Zhouhu Lan, Yngmn Tang, and Janguo Xao Insttute of Computer Scence and Technology, Pekng Unversty Abstract. Font recognton s an mportant and challengng problem n areas of Document Analyss, Pattern Recognton and Computer Vson. In ths paper, we try to handle a tougher task that ams to accurately recognze the font styles of texts n natural mages by proposng a novel method based on deep learnng and transfer learnng. Major contrbutons of ths paper are threefold: Frst, we develop a fast and scalable system to synthesze huge amounts of natural mages contanng texts n varous fonts and styles, whch are then utlzed to tran the deep neural network for font recognton. Second, we desgn a transfer learnng scheme to allevate the doman msmatch between synthetc and realworld text mages. Thus, large numbers of unlabeled text mages can be adopted to markedly enhance the dscrmnaton and robustness of our font classfer. Thrd, we buld a benchmarkng database whch conssts of numerous labeled natural mages contanng Chnese characters n 48 fonts. As far as we know, t s the frst publcly-avalable dataset for font recognton of Chnese characters n natural mages. 1 Introducton Font recognton s an mportant and challengng problem n areas of Document Analyss, Pattern Recognton and Computer Vson. Automatc font recognton can greatly mprove the effcency of many people s work. Frst and foremost, t helps people (not lmted to desgners) to know what ther favorte font styles are n the text n mages they see. Besdes, font producers employ t to fnd copyrght nfrngements by automatc font dentfcaton. Moreover, font recognton s useful n mprovng the accuracy and tme of optcal text recognton. Actually, t s a specfc problem of object detecton and classfcaton, n whch deep neural networks [4,5,9,13,14] have made great success. As we know, methods based on deep neural network demand large-scale tranng data and thus tme-consumng manual labelng s typcally requred. Unlke other object detecton and classfcaton tasks, the real-world font annotatons are extremely hard to get because large numbers of experts are needed to dentfy the fonts of texts n mages. Ths problem can be resolved to some extends by syntheszng hgh-qualty mages wth texts n dfferent fonts. However, there stll exst doman msmatch problems between syntheszed and real-world text mages. In Correspondng author. E-mal: lanzhouhu@pku.edu.cn

2 2 ths paper, we put emphass on how to synthesze hgh-qualty text mages of dfferent fonts and how to conduct effectve learnng from massve unlabeled mages wthout supervson. Up to now, many algorthms have been proposed for font recognton, such as modfed quadratc dscrmnant functons (MQDF) [8], wavelet feature descrptors [3], the texture descrptor based on fractal geometry [11], Gaussan mxture models [15], local bnary patterns (LBPs) [17], local feature embeddng (LFE) [2] and sparse features [16] etc. However, these tradtonal methods based on handcrafted features are not able to satsfactorly deal wth nosy data. Recently, neurodynamc models have been presented for solvng font recognton problem. The DeepFont system proposed n [20] employs a Convolutonal Neural Network (CNN) archtecture to recognze the font of Englsh text lnes. In addton to synthetc data augmentaton, a Stacked Convolutonal Auto-Encoder (SCAE) traned wth unlabeled real-world text mages s also utlzed to reduce overfttng. Another system reported n [18] s specfcally desgned to handle the Chnese Character Font Recognton (CCFR) task. They consdered CCFR as a sequence classfcaton problem and developed a 2-D long short-term memory (2DLSTM) model to capture a character s trajectory and dentfy ts font style. Although these recently-developed methods could markedly outperform tradtonal methods, they also have ther own shortcomngs. For nstance, the system proposed n [20] can only deal wth alphabetc language systems, such as Englsh, that consst of small number of dfferent characters. For heroglyph lke Chnese wth more than 6000 dfferent characters whose geometrc structures are often qute complcated, through experments we found that the SCAE does not work well. For real-world mages, the method developed by Tao et. al [18] may fal to capture characters trajectores especally when they are under complcated background and appear wth varous specal effects. Based on above-mentoned reasons, we select Chnese as one of the text languages n our experment and propose a transfer learnng algorthm to make use of unlabeled text mages. Also, our system ams to recognze the font of texts n natural mages, nstead of synthetc text mages adopted n [18]. Moreover, our method can be appled to any other language systems. Experments conducted on publcly-avalable databases demonstrate the effectveness of our system for font recognton n natural mages. 2 Overvew of the System As shown n Fgure 1, the proposed font recognton system can be bult as follows. Frst, we employ our engne to synthesze huge amounts of natural mages contanng texts n varous fonts and styles and meanwhle nformaton of each text lne s font and locaton s also recorded. Then, by usng the locaton nformaton we tran a text localzer to automatcally detect texts n mages collected from nternet. Thus, we have both labeled synthetc text mages and unlabeled real-world text mages. Afterwards, our ntal font classfer base on Convolutonal Neural Networks (CNNs) can be obtaned by tranng on labeled synthetc

3 3 Fg. 1. The ppelne of our font recognton system. text mages. Fnally, the proposed transfer learnng algorthm s mplemented to make use of unlabeled data and mprove the classfer s performance on recognzng fonts of texts n natural mages. Detals of each step n our system wll be explaned explctly n the followng sectons. 3 Syntheszng Text Images The synthetc text mage datasets such as the one descrbed n [20] only contan word-level mage regons and smple backgrounds. Thus they are unsutable to tran text detectors and font classfers for natural mages. The method proposed n [6] for generatng synthetc text mages naturally blends texts n exstng natural scenes, usng off-the-shelf deep learnng and segmentaton technques to algn texts to the geometry of a background mage and respect scene boundares. Inspred by the dea of ths method, we develop a new system to generate synthetc text mages wth texts n dfferent fonts and styles n cluttered condtons. As long as we get the TTF (True Type Font) or OTF (Open Type Font) fles of some fonts, we can generate nearly real text mages n these fonts automatcally, along wth the font label and locaton of each character. Fg. 2. Man steps to blend texts nto a background mage.

4 4 3.1 Blendng Texts nto Images As shown n Fgure 2, texts can be naturally blended nto a gven mage by usng our system. As we know, texts tend to be contaned n well-defned regons n realworld mages, nstead of crossng strong mage dscontnutes. For ths reason, we segment the mage nto contguous regons based on the cues of local color and texture nformaton usng the approach presented n [1]. After obtanng segmentaton regons, we choose sutable canddates from them for placng texts. Sutable regons should not be too small, should not have an extreme aspect rato, or have surface normal orthogonal to the vew drecton. In natural mages, texts are typcally panted on top of surfaces. In order to acheve a smlar effect n our synthetc data, we need to calculate the local surface normal of the regon where we are gong to put the text. To get the local surface normal of each contguous regon, we need to obtan an dense pxel-wse depth map, whch can be estmated by the CNN model proposed n [10]. Next, the text sample s assgned wth a color and a font. The font s chosen randomly from a font lst. Wth ths font s lbrary fle we can render the glyphs of text. Then the text s assgned wth a color whch matches well wth the background color. Fnally, the text s transformed accordng to the local surface orentaton and s blended nto the scene usng Posson mage edtng [12]. Note that, here we record the font type for each character nstead of ts content. In our experment, the text s font s chosen randomly from one of the fonts mentoned n Secton 6. Beyond that, we ntroduce more data augmentatons to sngle character mages to make them possess more vsually smlar appearance to real-world data. Detals are dscussed n the followng secton. 3.2 Text and Image Source We employ news corpora of dfferent languages, ncludng Arabc, Bangla, Chnese, Japanese, Korean and Englsh, as our text source. Each tme we randomly select some words from the corpus and blend them nto a background mage. The background mage for blendng text can nether be too smple nor too complex. To cover common scenes n our daly lves, we select mages from Open Images, an open dataset wth 9 mllon URLs to mages that were uploaded by users and have been annotated wth labels spannng over 6000 categores. We pck about 30,000 mages from the dataset wth the labels such as person and natural scenes nstead of street etc. whose backgrounds are too cluttered. 3.3 Data Augmentaton The synthetc texts generated by the above method are panted on flat surfaces and match well wth the background color. As a matter of fact, texts photographed n natural scenes may be blurred or n uneven llumnaton. To ncrease the dversty of our synthetc dataset, we apply some augmentaton processng to those synthetc text mages, ncludng rotaton, changng contrast and brghtness, GaussanBlur, addng Gaussan nose, and shear. The parameter of

5 5 each effect s selected randomly wthn certan range and these effects are stacked on one mage. Fg. 3. Texts n dfferent languages blended nto natural mages. Fg. 4. Blendng Chnese texts n dfferent fonts nto natural mages. 4 Text Detecton and Font Recognton To recognze the fonts of texts n natural mages, we frst need to accurately localze texts and then correctly dentfy ts font style. Snce text spottng technques have been extensvely studed n the last few years [6, 7, 19], our work does not focus on ths problem. The synthetc mages wth text locaton labels are utlzed to tran a CTPN (Connectonst Text Proposal Network) [19] as our text localzer, whch detects text lnes by fndng and groupng a sequence of fne-scale text proposals. The font recognton methods employed n ths paper are patch-based CNN models, the same as [20]. We frst extract square patches from a word mage, and send them to the convolutonal neural network. Each extracted patch, whose sde length equals the word mage s heght, contans one or more characters from the word mage. For each patch, the network outputs a vector wth each element correspondng to the probablty t belongs to each font. we average all vectors to determne the fnal classfcaton result of the word mage. Our font classfers are constructed by modfyng two famous CNN models (see Fgure 5),.e., AlexNet and VGG16, proposed n [9] and [14], respectvely.

6 6 Fg. 5. An llustraton of two modfed CNN models for font recognton. A 108*108 mage patch s put nto a network and the network ouputs a m-dmensonal probablty vector (m represents the number of font classes). AlexNet s a lghtweght network wth fast tranng speed whle VGG16 s a much deeper and more complex network. As we can see, VGG16 has more convolutonal layers and smaller convoluton strdes than AlexNet. 5 Boostng Accuracy va Transfer Learnng Due to the doman msmatch between synthetc data and real-world data, there stll exst a lot of text mages that can not be classfed correctly. Thus, we want to explot more nformaton from unlabeled real-world text mages. Fortunately, real-world text mages are easy to obtan from nternet. For example, when we type keywords lke text mages n a search engne, we can get large numbers of mages wth texts n varous knds of fonts. The proposed transfer learnng algorthm ams to further mprove the performance of font recognton for real-world text mages by makng use of the knowledge we gan from the syntheszed data. The key dea of our method s to try to assgn the unlabeled text mages wth correct tags by usng our ntal font classfers. Then, these newly-labeled texts mages can be adopted wth our synthetc mages to tran our CNN-based classfers agan to make them more robust and effectve. It s worthy of note that how we label text mages whose font categores are out of the range we consder. Snce some of them have smlar font styles, we should label them wth the most smlar fonts ncluded n our font lst. Meanwhle, we dscard those mages whose font styles are very dfferent aganst the fonts that we are nterested n. The problem to be addressed can be formulated as follows. Assume that the unlabeled dataset s composed of n text lnes (a text lne contans one or more words), let x be the th text lne of our dataset. The text lne x contans t () extracted patches and each patch n x s denoted by x j (1 n, 1 j t ()).

7 7 Our m-class font classfer takes a sngle patch as nput. After feedng a patch x j nto our pre-traned classfer, we get the probablty dstrbuton P ( xj) = ( ( ) ( ) ( )) ( ) P1 x j, P2 x j,..., Pm x j, n whch Pk x j means the probablty of x j belongng to the font f k (1 k m), and m k=1 P ( ) k x j = 1. The classfcaton result of x j s L ( ) ( ) p x j = arg maxk P k x j. For a gven text lne, we ntend to predct each patch s most probable font or dscard t based on the abovementoned analyses. Intutvely, the font type of patch mage x j can be labeled accordng to the ( ) ( ) classfcaton result L p x j. If PLp(x x j) j s smaller than a threshold, we dscard ths patch. However, our pre-traned CNN classfer may make mstakes when handlng real-world mages t has never seen before. If we label these unknown mages wth naccurate classfcaton results, the classfer would be ncorrectly supervsed whch often results n a declne n ts performance. Typcally, text mages possess a property that characters (or words) n one text lne are usually n the same font style. On account of ths, the font labels of patches n one text lne dentfed by the classfer are supposed to be dentcal. If the labels are not dentcal, t means that our pre-traned classfer fals to adapt new data. Through a statstcal analyss of all patches classfcaton results n the same text lne, we select a representatve font style to relabel them. In ths manner, the probablty of makng mstake can be greatly reduced. Our method s desgned as follows: for each font f k (1 k m), we defne two varables to estmate how lkely ths entre text lne x s n font f k. The frst varable s A (k) = t() j=1 1 { ( ) } L p x j = k, meanng the tmes that label k appears n the predcted labels of patches n x. The other one s B (k) = t() j=1 P ( ) k x j, denotng the probablty of x belongng to font f k accumulated by patches n the text lne. We use A as the frst sort key and B as the second sort key to rank these fonts (f 1, f 2,..., f m ) (f k ranks ahead f A (k) or B (k) s hgher). Let the font rankng frst be f l, f B (l) th t () (th s set to 0.4 here), we assgn the label of each patch n text lne x wth f l. The method s actually a votng procedure to decde a text lne s font. As a general rule, n turn-based games who wnng more rounds wns the game. The pseudo code of ths method s shown n Algorthm 1. 6 Experments 6.1 Font Recognton of Chnese Text Images To measure the performance of our font recognton method, we need to collect real-world text mages contanng characters n varous font styles as our test dataset. We cooperate wth Founder Electroncs, one of the world s largest Chnese font producer, to buld a large-scale database for font recognton n natural mages, named VFRWld-CHS 1. Specfcally, the VFRWld-CHS dataset conssts of 816 text mages captured n natural scenes, from whch 6,827 sngle 1

8 8 Algorthm 1 Predctng Labels for Unlabeled Text Images Input The font label lst (f 1, f 2,..., f m), the unlabeled patches set T = { x j 1 n, 1 j t () }, the pre-traned font classfcaton functon P. Output The most probable font L ( xj) for each patch x j. 1: for = 1 n do 2: for ( j = ( 1 ) t () ( do ) ( )) ( ) 3: P1 x j, P2 x j,..., Pm x j P x ( ) ( ) j 4: L p x j arg maxk P k x j 5: end for 6: for k = 1 m do 7: A (k) t() j=1 1 { ( ) } L p x j = k 8: B (k) t() j=1 P ( ) k x j 9: end for 10: sortedlst sort(f 1, f 2,..., f m), sortkey(a, B) 11: f l sortedlst[0] 12: f B (l) th t () then 13: for j = 1 t () do 14: L ( xj) fl 15: end for 16: end f 17: end for Chnese character mages n 48 fonts are extracted and labeled. As we can see from Fgure 6, the nosy backgrounds and specal effects added artfcally make t qute dffcult to locate the characters and recognze ther font styles. We prepare three dfferent datasets to tran our font classfer. The frst dataset, denoted as Syn Smple, conssts of smple synthetc character mages wth no augmentatons (black characters rendered n whte background mages). The second dataset, denoted as Syn Blend, s composed of sngle Chnese character mages cropped from synthetc mages generated by our text mage syntheszng method wthout data augmentatons. Sample mages of these two datasets are shown n fgure 7. We apply the augmentaton methods mentoned n Secton 3.3 to Syn Blend and get a larger dataset Syn Blend Aug. Detaled nformaton of these datasets s descrbed n Table 1. Besdes, we buld a database consstng of more than 200,000 unlabeled mages whch are collected from nternet (See Fgure 8). We compare the performance of our classfers on the test set whch are traned on the above-mentoned tranng datasets, respectvely. Through our experments, we fnd that compared to Syn Smple, the synthetc text mages generated by our method can sgnfcantly mprove the classfcaton performance on our test dataset. As t can be observed from Table 2, the accuracy of our classfer based on AlexNet and VGG16 s very low when traned wth Syn Smple, but mproves consderably when traned wth Syn Blend and Syn Blend Aug. The result s reasonable because the mages n Syn Blend and Syn Blend Aug look more natural than mages n Syn Smple. To sum up, the method mentoned n Secton 3 s an effectve soluton to recognze fonts of texts n natural mages.

9 9 Fg. 6. Examples of sngle Chnese character mages cropped from text mages n our test dataset. Fg. 7. The Left mages (n Dataset Syn Blend) are synthetc characters generated by our method. The rght mages (n Dataset Syn Smple) are correspondng blankbackground and no-specal-effect characters. Fg. 8. Some text lnes detected by the our text localzer. These mages come from Internet and have some specal effects and manual desgns our synthetc mages don t have.

10 10 Table 1. Comparson of all datasets adopted n our experment name Source Label? Purpose Sze Class VFRWld-CHS Real Y Test 6, Syn Smple Syn Y Tran 324, Syn Blend Syn Y Tran 474, Syn Blend Aug Syn Y Tran 670, Unlabeled Dataset Real N Tran 229,044 N/A 1 The unlabeled dataset conssts of text lne mages. The others consst of sngle Chnese character mages. Next, we would lke to verfy the effectveness of our transfer learnng scheme. As shown n Table 2, our transfer learnng algorthm further mproves the classfer s performance. On the contrary, f we drectly label each character wth the predcted result gven by the ntal classfers (dscard t f the classfcaton probablty s lower than th), we wtness a declne n classfcaton accuracy: AlexNet top %, VGG16 top % n our experment. Ths demonstrates the effectveness of the proposed transfer learnng scheme n font recognton tasks. Table 2. Our method s performance on VFRWld-CHS Accuracy Method Model SS SB SBA TL AlexNet(top-1) 13.85% 69.30% 71.14% 77.75% AlexNet(top-5) 46.80% 90.75% 91.12% 93.93% VGG16(top-1) 34.21% 81.93% 84.83% 87.68% VGG16(top-5) 53.68% 95.22% 96.14% 97.53% 1 SS, SB and SBA denote our proposed methods traned on Syn Smple, Syn Blend and Syn Blend Aug datasets, respectvely. TL denotes transfer learnng. 6.2 Comparson wth Other Methods We compare the performance of our methods wth other recently-proposed approaches on VFRWld-CHS. LFE (local feature embeddng) ntroduced n [2] s a representatve tradtonal method whch fuses handcrafted local features. DeepFont F ntroduced n [20] uses synthetc text mages wth tradtonal augmentatons to tran a convolutonal neural network. These two methods, along wth our SBA method, are supervsed learnng methods. DeepFont CAEFR [20] and our transfer learnng method are both sem-supervsed methods explotng unlabeled real-world mages. For comparatve analyss, we employ the same network archtecture as [20], whch s very smlar to AlexNet. The dfference s that we utlze meth-

11 11 ods ntroduced n Secton 3 and 5 to synthesze tranng data and explot unlabeled data. As shown n Table 3, our method outperforms other methods. The VFRWld-CHS dataset features nosy backgrounds and dstortons, whch can not be properly handled by methods of [2] and [20]. Results shown here verfy the effectveness and generalty of our methods. Table 3. Comparson of dfferent methods performance on VFRWld-CHS Accuracy Methods TOP-1 TOP-5 LFE [2] 32.65% 60.69% DeepFont F [20] 50.26% 72.93% SBA(ours) 70.97% 91.05% DeepFont CAEFR [20] 55.58% 76.21% TL(ours) 77.68% 93.97% 7 Concluson In ths paper, we developed a new system for accurate font recognton n natural mages. One major advantage of our system s that tme-consumng and costly font annotatons for mages n the tranng dataset can be avoded. On the one hand, by blendng text nto background mages and mplementng data augmentatons, the syntheszed text mages look more real and thus large-scale hgh-qualty tranng data can be automatcally constructed for our CNN based font classfers. On the other hand, the ntroducton of our transfer learnng algorthm explots a large corpus of unlabeled real-world mages and thereby sgnfcantly mproves the capacty and accuracy of classfcaton. Expermental results on a publcly-avalable database we bult demonstrated that consderable good performance of font recognton n natural mages can be obtaned by usng our system. References 1. Arbeláez, P., Pont-Tuset, J., Barron, J.T., Marques, F., Malk, J.: Multscale combnatoral groupng. In: Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton. pp (2014) 2. Chen, G., Yang, J., Jn, H., Brandt, J., Shechtman, E., Agarwala, A., Han, T.X.: Large-scale vsual font recognton. In: Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton. pp (2014) 3. Dng, X., Chen, L., Wu, T.: Character ndependent font recognton on a sngle chnese character. IEEE Transactons on pattern analyss and machne ntellgence 29(2), (2007) 4. Grshck, R.: Fast r-cnn. In: Proceedngs of the IEEE Internatonal Conference on Computer Vson. pp (2015)

12 12 5. Grshck, R., Donahue, J., Darrell, T., Malk, J.: Rch feature herarches for accurate object detecton and semantc segmentaton. In: Proceedngs of the IEEE conference on computer vson and pattern recognton. pp (2014) 6. Gupta, A., Vedald, A., Zsserman, A.: Synthetc data for text localsaton n natural mages. In: Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton. pp (2016) 7. Jaderberg, M., Smonyan, K., Vedald, A., Zsserman, A.: Readng text n the wld wth convolutonal neural networks. Internatonal Journal of Computer Vson 116(1), 1 20 (2016) 8. Kmura, F., Takashna, K., Tsuruoka, S., Myake, Y.: Modfed quadratc dscrmnant functons and the applcaton to chnese character recognton. IEEE Transactons on Pattern Analyss and Machne Intellgence (1), (1987) 9. Krzhevsky, A., Sutskever, I., Hnton, G.E.: Imagenet classfcaton wth deep convolutonal neural networks. In: Advances n neural nformaton processng systems. pp (2012) 10. Lu, F., Shen, C., Ln, G.: Deep convolutonal neural felds for depth estmaton from a sngle mage. In: Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton. pp (2015) 11. Moussa, S.B., Zahour, A., Benabdelhafd, A., Alm, A.M.: New features usng fractal mult-dmensons for generalzed arabc font recognton. Pattern Recognton Letters 31(5), (2010) 12. Pérez, P., Gangnet, M., Blake, A.: Posson mage edtng. In: ACM Transactons on Graphcs (TOG). vol. 22, pp ACM (2003) 13. Ren, S., He, K., Grshck, R., Sun, J.: Faster r-cnn: Towards real-tme object detecton wth regon proposal networks. In: Advances n neural nformaton processng systems. pp (2015) 14. Smonyan, K., Zsserman, A.: Very deep convolutonal networks for large-scale mage recognton. Computer Scence (2014) 15. Slmane, F., Kanoun, S., Hennebert, J., Alm, A.M., Ingold, R.: A study on fontfamly and font-sze recognton appled to arabc word mages at ultra-low resoluton. Pattern Recognton Letters 34(2), (2013) 16. Song, W., Lan, Z., Tang, Y., Xao, J.: Content-ndependent font recognton on a sngle chnese character usng sparse representaton. In: Document Analyss and Recognton (ICDAR), th Internatonal Conference on. pp IEEE (2015) 17. Tao, D., Jn, L., Zhang, S., Yang, Z., Wang, Y.: Sparse dscrmnatve nformaton preservaton for chnese character font categorzaton. Neurocomputng 129, (2014) 18. Tao, D., Ln, X., Jn, L., L, X.: Prncpal component 2-d long short-term memory for font recognton on sngle chnese characters. IEEE transactons on cybernetcs 46(3), (2016) 19. Tan, Z., Huang, W., He, T., He, P., Qao, Y.: Detectng text n natural mage wth connectonst text proposal network. In: European Conference on Computer Vson. pp Sprnger (2016) 20. Wang, Z., Yang, J., Jn, H., Shechtman, E., Agarwala, A., Brandt, J., Huang, T.S.: Deepfont: Identfy your font from an mage. In: Proceedngs of the 23rd ACM nternatonal conference on Multmeda. pp ACM (2015)