Projet ANR- Blanc International II - SIMI 2 - Science informatique et applications GUWENSHIBIE. Programme Blanc International II 2012

Transcription

1 Projet ANR- Blanc Internatonal II - SIMI 2 - Scence nformatque et applcatons GUWENSHIBIE Programme Blanc Internatonal II 2012 Work Package n 2 Delverables n 4 and n 5 Image Pre-processng and Restoraton Benchmark and evaluaton of restoraton tools Self controlled algorthms to tune automatcally the parameters of restoraton and enhancement. Author: Le Bourgeos Frank Delvery date: 01 November 2014

2 1. Objectves Among ancent Chnese manuscrpts, some of them show defects due to the agng of paper and nk or due to dgtzaton constrants. Image pre-processng conssts of enhancng the mages n order to smplfy the layout analyss, the extracton of robust features and the strokes retreval, whch s essental to Optcal Character Recognton. In the opposte, the mage restoraton conssts of retrevng the exact nformaton lost because of the nk and paper agng. Image restoraton s dedcated to hstorans, palaeographers, codcologst and users who want to study the document n the orgnal layout. For the Guwensbe project, we have two objectves : Dgtally enhance the mage by pre-processng algorthms n order to smplfy the mage recognton steps. Try to segment the mage to separate character shapes from the paper wthout loss of nformaton about the character patterns. 2. Study of the type of degradatons on Chnese rolls Chnese manuscrpts show specfc degradatons that we have classfed nto three categores : 2.1 Paper Degradatons The paper of Ancent manuscrpts show many defects whch appear as nose n the background of the mages. It s the man degradaton observed n these manuscrpts. It leads to ghost patterns n the background of the dgtal mage and colour nose whch can be confused wth the colour of the characters. Chnese manuscrpts have stans due to the humdty whch must be removed or reduced n order to make the character segmentaton easer. Ancent Chnese manuscrpts can present some holes or mssng parts whch cannot be flled by dgtal npantng f the sze of the hole s too large. Dgtal npantng can fll thn holes lke scratches. Generally, ancent Chnese manuscrpts show large holes whch cannot be restored by dgtal npantng. Unlke Latn or Arabc manuscrpts, Chnese ancent rolls are handwrtten mostly n only one sde. Consequently, we rarely notce nk bleed-through degradaton when the nk can bleed through to the other sde so that the characters from the reverse sde appear on the foreground. 2.2 Ink Degradatons These degradatons can lead to broken or touchng characters, whch are the major causes of OCR errors. The nk fadng and dsappearance s the most frequent degradaton observed on Chnese manuscrpts. Ths loss of nformaton s caused by the agng of nks. Once the document s dgtzed, these degradatons become a part of the document. The nk fadng leads to a complete or partal loss of the nformaton

3 n some parts of the characters because the nk has the same colour as that of the paper. It decreases the performance of the OCR and makes more dffcult the segmentaton of characters. Restorng the nk fadng conssts of enhancng the remanng nformaton about parts of characters whch are no more vsble and to extrapolate lost nformaton. The degradaton of nks due to agng also mpacts the qualty of the dgtal mages of the characters. The nose along the contours of the characters and also nsde the characters must be reduced n order to facltate the segmentaton process and the feature extracton. 2.3 Degradaton due to dgtzaton constrants These degradatons appear durng the dgtzaton process and depend on the dgtzer and the work of the operator. For the Guwenshbe project, the dgtzaton process s nearly optmal. The rolls are perfectly algned, and the mage does not requre any deskew operaton. The manuscrpts are mantaned flat so that there are no geometrcal dstortons lke curvatures, or warpng. Illumnaton s unform along all the rolls. There s no pre-processng operaton to restore degradatons due to the dgtzaton process. 2.4 Summary of degradatons Table 1 summarzes the dfferent types of degradaton oberved n Chnese manuscrpts and fgure 1 s a good llustraton of these defects. We do not develop pre-processng tools for the degradatons whch are not observed nto Chnese manuscrpts of the project. Fgure 1: Frequent degradatons observed on Chnese hstorcal manuscrpts.

4 Degradaton Soluton Degradatons of the papers Nose n the background Paper stans Mssng parts nk bleed-through Denosng by PDE regularzaton lke TV, ansotropc dffuson, Non Local Means, Blateral flterng... Colour analyss by morphology, colour clusterng, colour enhancement No attempt to fll holes by dgtal npantng technques or by sparse analyss because we know n advance that these approaches fal when the holes are too large. Too much nformaton s lost for a reconstructon.. No such degradaton observed Degradatons of the nk The nk fadng and the nk dsappearance Colour analyss by morphology, colour clusterng, colour enhancement, Retnex, Ink noses Denosng by PDE regularzaton lke TV, ansotropc dffuson, Non Local Means, Blateral flterng, Coherence-Enhancng flterng Degradaton due to dgtzaton constrants Non llumnaton and no colour constancy No such degradaton observed Out of focus blurrng whch requre mage No such degradaton observed sharpenng, chockng or deblurrng Geometrcal degradatons (paper No such degradaton observed warpng, skew, rotaton, curvature...) Table 1: Lst of the degradatons and the assocated algorthms to enhance the mage. In concluson, the man pre-processng operatons to experment are the mage denosng and colour segmentaton or enhancement n order to separate the Chnese characters around stans or when the nk s fadng. We favour methods whch are not based on machne learnng and whch do not requre pror nformaton or a model because such approaches can fal on handwrtten manuscrpts wth complex shapes havng too many spatal varatons.

5 3. Expermentaton on mages of Chnese manuscrpts The objectve conssts to develop an autonomous system for mage restoraton and enhancement already developed n the LIRIS lab. We have tested all exsted on dgtzed Chnese manuscrpts. We must frst benchmark and evaluate exstng tools developed n the LIRIS and n the lterature, compare the results and fnd the default parameters for each method. We also propose to automatcally tune the parameters accordng to the mage contents when t s feasble. We apply the algorthms on dfferent mages from IDP webste showng severe degradatons. As the sze of the mages are very large, we just llustrate n ths report, the effects of each algorthm on a very small part of the orgnal mage, at the orgnal scale presented n Fgure 2. Fgure 2: Small part of the orgnal Image zoomed at 100%. Ths llustraton shows the typcal degradatons due to the nose of the nk and the background and the nk fadng.

6 3.1 Denosng of nk and background paper We do not propose approaches whch requre pror nformaton or based on machne learnng whch requre tranng. Model drven restoraton technques are effcent only f the degradatons can be observed statstcally several tmes n the same confguraton. In ths case we can tran a Neural Network or buld dctonares wth a sparse analyss to learn repeated degradatons for each confguraton of pxels and ts neghbourhood. The model drven approach works perfectly well for prnted characters havng predctable and repettve structures whch can be learned. In the case of handwrtten manuscrpts, the spatal varablty s too hgh and the number of possble local structures s too great that t s dffcult to tran a system or fnd statstcal repettons to buld dctonares. We focus only on data drven approaches whch do not requre any tranng or pror knowledge on character patterns. We detal three dfferent pxel-drven approaches to reduce nose: Weghted Averagng approaches Approaches based on Total Varaton Approaches based on Heat equaton Most of these approaches can be solved by varatonal methods and expressed mathematcally by Partal Dfferental Equatons of PDE Weghted Averagng models These approaches use a weghted averagng wth pxels x r found locally n the neghbourhood of the central pxel x r or non locally n smlar parts of the mage (1). The weghts w( x r ) can use spatal nformaton or/and colour nformaton n the colour space. (1) r r x w r x = r w ( x ) ( x ) Dfferent models use the weghted averagng approach to reduce nose and segment the mage lke the Blateral flter, Non Local Mean and the MeanShft. Accordng to the weghts calculaton and the nformaton used for the averagng, the weghted averagng produces completely dfferent results.

7 a) Blateral flter A blateral flter s a non-lnear edge-preservng and nose-reducng smoothng flter for mages. The ntensty value at each pxel n an mage s replaced by a weghted average of ntensty values from nearby pxels havng smlar colours (2). Theses weghts depend on the Eucldean dstance of pxels from the central pxel and on the colour dstance n the colour space. A Gaussan dstrbuton s used to adjust the computaton of the weghts. The standard devatons σ R and σ S adjust the weghts for the range colour dstance n the colour space and the Eucldean dstance n the spatal space, respectvely [Tomas98]. It smultaneously sharps characters contours and reduce the nose of the background. (2) r x p = r x K K r r σ ( x x ) K ( p) R p σ S r r ( x x ) K ( p) p σ S Fgure 3: Applcaton of the Blateral flterng The blateral flterng does not smooth the nose along character contours and cannot repar the dscontnutes of broken characters. However, t sharps the character contours. Ths flter s not advsed to restore mages of ancent manuscrpts because of the loss of nformaton and detals necessary for the OCR.

8 b) Non Local Means of Buades [Buades05] ntroduce the Non Local means whch use the mage redundances to reduce nose. The ntensty value at each pxel s replaced by a weghted average of ntensty values from neghbours havng smlar neghbourhoods (3). It s smlar to the blateral flterng but wth non local nformaton found n neghbourhoods havng smlar contents. The weghts of the averagng depend on the smlarty of the contents between the neghbourhood N(x ) and the local nformaton N(x). The standard devatons σ R adjust the flterng. The hgher σ R s, the stronger the flterng s. r r r x Kσ ( N ( x ) N ( x) ) R r (3) x = r r K N x N x σ R ( ( ) ( )) Fgure 4: Applcaton of the Non Local Means Compare to the Blateral flter, the NLM performs better. However, for ancent manuscrpts wth complex and non repettve structures, the NLM fals to fnd smlar neghbourhoods. If the NLM does not fnd smlar neghbourhood locally, we automatcally enlarge the search wndow to a larger area. The best soluton wll be to search smlar neghbourhoods n the whole mage. However, n ths case, the computaton complexty wll be too hgh to process an entre mage n reasonable tme. Lke the Blateral, the NLM fals to repar dscontnutes when the nk s fadng. Furthermore, the NLM smooth the contours and preserve all the detals necessary for the character recognton.

9 3.1.2 Total Varaton The Total Varaton or TV from Rudn, Osher, Fatem [Rudn92] mnmzes the ntegral of the magntude of the gradent of the sgnal and removes unwanted detals lke nose and speckles whle preservng mportant detals such as edges. It corresponds to the applcaton of the followng PDE (4). We use the splt Bregman optmzaton to speed up the mage processng. Mn Ω I I I dxdy (4) = dv t I Fgure 5 Effect of the TV wth splt Bregman optmzaton Lttle and thn detals of the Chnese characters are not preserved by the TV flterng. Dscontnutes of strokes are not restored by the TV. Ths model also fades the nk and enlarges the stroke dscontnutes.

10 3.1.3 Ansotropc Dffuson Ansotropc dffuson mnmzes the ntegral of the square of the magntude of the gradent of the sgnal. The Soluton leads to PDE (5) for a scalar dffusvty coeffcent and (6) for a matrx dffusvty coeffcent. Mn I dxdy (5) (6) = dv( D I ) a) Perona & Malk scalar drven dffuson The scalar drven dffuson (7), ntroduced by [Perona90], preserves the edges but does not smooth the contours of the characters. The coeffcent of dffusvty s a decreasng curve. It stops the dffuson when the magntude of the gradent s larger than a predefned threshold K. The parameter K delmts the dfferences between the nose and the pxels of the contours. We expect that nose have a lower magntude than the contours of characters. I t Ω (7) = dv( c( I ) I ) 2 I = dv( c I ) t C I I t I = exp ² wth ( ) K² Fgure 6: Scalar drven dffuson of Perona&Malk wth K=10 Nose and defects reman along the contours and broken characters are not repared.

11 b) Coherence enhancng ansotropc dffuson from Weckert [Weckert96] ntroduces a new model whch renforces the coherency of the contnuty of the strokes. He was the frst to ntroduce the matrx drven dffuson nto the heat equaton. The matrx D of the dffusvty s calculated from the smoothed tensors feld. It s orented to the man drecton of the tensors and D s always ansotropc along ths drecton (8). T T I D = µ + ΘΘ + µ Θ+ Θ+ (8) = dv( D I ) t ρ Θ+ /, λ+ / egen vectors and values of Tensors Tσ ρ T Tensors felds smoothed by Gaussans Tσ = Gρ ( ( Gσ I) ( Gσ I) ) The parameters of the Weckert model are C µ ( ) + = α + 1 α exp ( ) f λ+ λ 2 λ+ λ and µ + = α else µ = α Fgure 7: Matrx drven dffuson of Weckert As expected, the contnutes of lnes are renforced, but the endngs of the lnes may change of drecton. Important local nformaton lke ntersectons and mcro structures are damaged by ths model of restoraton.

12 c) D. Tschumperle Trace of the Hessan. [Tschumperle02] ntroduces a more effcent model of denosng based on the trace of the Hessan Matrx (9). The effcency of ths model has been proven. It can restore heavly degraded natural mages. 1 T 1 T I D = ΘΘ + Θ+ Θ+ (9) = trace( DH ) 1+ λ λ 1 λ+ λ t ρ Θ+ /, λ+ / egen vectors and values of Tσ Fgure 8: Ansotropc dffuson of D. Tschumperle Appled on Chnese ancent manuscrpts, ths model does not repar the dscontnutes of strokes, but t reduces effcently the nose. The man drawback of ths model s the loss of nformaton when the nk s fadng. However, ths model preserves all exstng nformaton wthn the mage.

13 d) Ansotropc dffuson whch preserves edges and sngulartes by the LIRIS Developed exclusvely for text denosng, ths model seeks to preserve all mportant nformaton for the OCR, especally sngulartes lke ntersectons, strokes endngs, junctons of strokes. It s a combnaton of two models, the Perona&Malk model whch preserves sngulartes and the Weckert coherency enhancement model whch restores the contnutes of lnes (10). The resultng model of ths combnaton s a new model whch preserves sngulartes and smultaneously renforces the contnutes of strokes [Drra11]. Two new parameters K+ and K- are necessary to fx. K+ defnes the contours to regularze and K- defnes the sngulartes to preserve. I t 10) = dv( D I ) Θ + /, λ D + / * = e λ K Θ + Θ T + egen vectors + e λ+ K+ Θ Θ T and values of T ρ σ Fgure 9: Ansotropc dffuson whch preserves sngulartes and renforces the contnutes of strokes wth K+ = 20 and K- = 100 Ths flter preserves the exstng nformaton requred by the OCR and reconstructs the contnutes of strokes. The man drawback s the parameters K+ and K- to fx manually or automatcally. The automatc calculaton of these parameters has been done for the entre mage. The parameters could fal f the background colour change locally around stans for example. Local adaptaton of these parameters stll does not work properly.

14 e) The Beltram flow Wth the edge and sngulartes preservng dffuson, the Beltram flow [Kmmel00] s among the best models to process mages of documents and especally ancent manuscrpts (11). It preserves all nformaton necessary for the OCR and reduces effcently the nose of the nk and the paper. It also renforces a lttle the contnuty of lnes. 1+ λ+ µ + = I 1 1+ λ (11) = dv( D I ) t ( 1+ λ + )( 1+ λ ) 1+ λ µ = 1+ λ+ Fgure 10: Ansotropc dffuson of Beltram Among all exstng models, the Beltram flow performs perfectly well. Among all ansotropc dffuson models presented prevously, t s the more effcent model. We advse to use the Beltram flow for the reducton of the nose when there are not too many dscontnutes. In the case of large dscontnutes, we advse to use the ansotropc dffuson whch preserve sngulartes and renforce the coherency of lnes.

15 3.2 Colour segmentaton and enhancement Our objectve conssts of testng the feasblty to segment correctly Chnese characters from degraded manuscrpts. The bnarzaton of the lumnance by any local or global thresoldng completely fals to retreve erased characters by nk fadng. We need to analyze the mage drectly n colour to keep all necessary nformaton for a correct segmentaton. Among all expermentaton, we detal only successful combnaton of methods whch provde the best results we obtan after several days of computaton to tune the optmal combnaton of methods and the optmal parameters The MeanShft The Orgnal MeanShft (MS) s a generc unsupervsed clusterng procedure based on maxma of denstes functon of colour clusters n the colour space [Fukunaga75,90]. The MS shfts teratvely each colour of the mage to the nearest mean along the gradent drectons of the densty functon n the colour space. It s a weghted averagng technque lke the blateral or the Non Local Mean. The weghts of the averagng are the dstance between colours n the colour space (12). The MS segmentaton process requres no explct defnton of the clusters and s based on the densty estmaton nto a Parzen-wndow whch makes t well-suted for dealng wth nosy and uncertan data sets. The orgnal MS can cluster correctly non Gaussan clusters lke colours n the colour space. (12) r x = r x r r x x r r x x ( x) = { x / d ( x x) σ } x N, The MS requres only one parameter whch s σ R the sze of the Parzen Wndow for a correct estmaton of the densty functons of colours n the colour space. The complexty of the MS s explaned by the ntensve search of colour samples n the Parzen wndow to compute the vector orented towards the mean. Consequently, the complexty of the MS s O(N²) wth N equal to the number of pxels of the mage. Wthout optmzaton the MS wll take several days to segment only one mage. Several attempts have tred to decrease the algorthm complexty manly by addng spatal nformaton or by reducng the number of colours to shft or even by selectng a reduced number of colours to estmate the means of the densty functon. But all these optmzaton ether degrade the results or do not reduce enough the processng tme. We decrease the complexty of the orgnal MS from O(N²) to O(N) by usng ntegral mages [LeBourgeos13]. A hgh resoluton mage of 5000x4000 can be processed n only few seconds compare to several days. The resultng fgure 11, show that the MS fals to retreve erased parts of characters when the nk s fadng. Pre-processng technques must be appled to enhance the fadng colours Mult-scale Retnex colour Enhancement (MSR) [Rahman96] has developed for the NASA, an effcent colour enhancement algorthm called Mult-Scale Retnex (MSR). We use such an algorthm to mprove the document legblty or to retreve colour nformaton (Fgure 12). c R

16 Fgure 11: Segmentaton by fast ntegral MeanShft wth σ R =7 Fgure 12: Color enhancement by Multscale Retnex (MSR)

17 3.2.3 The MeanShft on MSR mage If we compute the MeanShft segmentaton on MSR mages, we obtan a better segmentaton of characters. But some parts of several characters are stll mssng. The MSR also enhances the colormetrc nose whch has been segmented by the MS. It produces false detecton of patterns nto the background. Moreover, nose remans along the contours of the characters and nto the background. Fgure 13 MeanShft on MSR Image The MeanShft combned to anstropc dffuson on MSR mage We propose to combne ansotropc dffuson wth the MeanShft n order to smultaneously segment the mage and reduce the colormetrc nose. Such work has already been successfully expermented but not yet publshed. We use the ansotropc dffuson whch preserves edges and sngulartes from [Drra11] combned to the fast ntegral MeanShft n parallel. The result n fgure14 shows real mprovement of the qualty of the fnal segmentaton and the absolute lmt of the colour segmentaton by classcal methods. When the colour of the background changes, the classcal approach may fal to correctly segment the characters from stans. Fgure 15 presents the success of the colour segmentaton on characters wrtten n stans.

18 Fgure 14 : Combnaton of MS and ansotropc Dffuson on MSR mage Fgure 15 : Bnarzaton of the processed mage showng the character segmentaton

19 Orgnal mage wth stans Mult-Scale Retnex Correcton Combnaton Ansotropc dffuson+ms Bnary mages wth segmented characters Fgure 16 : Result of the combnaton of MS and ansotropc Dffuson on MSR mage wth characters wrtten nsde stans of humdty.

20 3.2.5 Segmentaton by Colour Morphology In the LIRIS laboratory, we have recently developed a colour segmentaton software called AColDPS (Robust and Unsupervsed Automatc Colour Document Processng System) to segment degraded busness documents. Ths work wll be publshed soon. Ths very recent work uses manly colour morphology and does not requre any tranng, manual assstance, pror knowledge or model. AColDPS processes the mages dfferently. It does not try to segment characters accordng to the colour nformaton. It segments, by colour morphology, only thn objects whatever ther colours. After the segmentaton of thn colour objects, AColDPS fnds the rght colour of all segmented object by usng colour nformaton from the MeanShft operaton. We have tested ths morphologcal process on several mages of Guwenshbe project. We obtan also good performance compare to the combnaton of the MeanShft and Dffuson on MSR mage. It processes text wrtten nto stans correctly and fnds the correct colour class of characters (Fgure17). Nose s also removed by morphologcal operatons and not by dffuson. Compare to prevous methods, the contours of characters are not smoothed, but the characters are well segmented. Fgure 17: segmentaton by AColDPS

21 3.3 Lmts of the segmentaton In spte of all advanced algorthms, character segmentaton can fal for strongly degraded manuscrpts when characters nk dsappear (Fgure18). Fgure 18: segmentaton by Combnaton of MS and ansotropc Dffuson on MSR mage There s no perfect segmentaton system whch can well segment all type of documents whatever the degradaton. It justfes the research n the other work packages on robust features and layout extracton whch do not requre character segmentaton, mage bnarzaton and connected components analyss. 3.4 Automatc tunng of parameters Most of the parameters are fxed by default to values found n the scentfc lterature. The others are found by expermentaton. Some crtcal parameters are tuned automatcally to optmally process mages correctly. The user can manually modfy some crtcal parameters lke the σ R of the MeanShft. 4. Software The software Docrestorer has been developed n C++, t can process, n a workflow, thousands of mages sequentally wth the same parameters. The selected operatons to apply are desgned through the user nterface. It wll be freely dstrbuted to all the partners of the Guwenshbe project for research purpose.

22 Fgure 19: screenshot of the software Docrestorer 5. Concluson We have expermented only pxel-based pre-processng approaches from the lterature and the man colour segmentaton methods exstng n the LIRIS laboratory. Most of the weghted averagng approaches do not reduce nose along the contours of the characters and cannot repar broken characters. The pre-processng operatons to reduce nose, work perfectly well wth varatonal methods lke the Beltram flow. But t can totally fal by usng the Total Varaton model. The segmentaton of characters by colour analyss or by morphology s feasble on degraded manuscrpts. For severe degradatons, when the nose magntude s greater than the magntude of the contours of the characters, the segmentaton of characters s dffcult or mpossble to acheve wthout pror knowledge. It s also the case when the nk dsappears. The dffcultes to segment correctly characters on degraded manuscrpts justfy the future research about new robust features and new layout extracton whch do not requre any segmentaton.

23 6. References [Buades05] Buades, Anton "A non-local algorthm for mage denosng". Computer Vson and Pattern Recognton, : [Drra11] F. Drra, F Le Bourgeos, A New PDE-based Approach for Sngularty- Preservng Regularzaton: Applcaton to Degraded Characters Restoraton, n Internatonal Journal on Document Analyss and Recognton (IJDAR) [Fukunaga90] K. Fukunaga, Introducton to Statstcal Pattern Recognton. Boston, MA: Academc Press, [Fukunaga75] K. Fukunaga and L.D. Hostetler, The estmaton of the gradent of a densty functon, wth applcatons n pattern recognton. IEEE Trans on Informaton Theory 21 (1975), [Kmmel00] R. Kmmel, R. Mallad, and N. Sochen. Images as embedded maps and mnmal surfaces: moves, color, texture, and volumetrc medcal mages. Internatonal Journal of Computer Vson, 39(2): , September [Lebourgeos13] F Lebourgeos, F Drra, DJ. Gaceb, J Duong, Fast Integral MeanShft : Applcaton to Color Segmentaton of Document Images. Dans Twelfth Internatonal Conference on Document Analyss and Recognton (ICDAR 2013), IEEE ed. Washngton, USA. pp [Perona90] P. Perona, J.Malk. Scale-Space and Edge Detecton Usng Ansotropc Dffuson, IEEE Trans. Pattern. Analyss and Machne Intellgence, pp , [Rahman96] Z. Rahman, D. J. Jobson, and G. A. Woodell, Multscale Retnex for Color Image Enhancement, Internatonal Conference on Image Processng (ICIP) '96. [Rudn92] Rudn, L. I.; Osher, S.; Fatem, E. (1992). "Nonlnear total varaton based nose removal algorthms". Physca D 60: [Tomas98] C. Tomas and R. Manduch. Blateral _lterng for gray and color mages. In Proceedngs of the IEEE Internatonal Conference on Computer Vson, pages , January [Tschumperle02] D. Tschumperlé, R. Derche. Dffuson PDE s on Vector-valued Images : Local Approach and Geometrc Vewpont. Dans IEEE Sgnal Processng Magazne, Vol 19. No 5. pp , [Weckert96] J. Weckert. Ansotropc Dffuson n Image Processng, Laboratory of Technomathematcs, PhD, Unversty of Kaserslautern, Germany, 1996.