Robust Vanishing Point Detection for MobileCam-Based Documents

Transcription

1 011 Internatonal Conference on Document Analss and Recognton Robust Vanshng Pont Detecton for MobleCam-Based Documents Xu-Cheng Yn, Hong-We Hao Department of Computer Scence School of Computer and Communcaton Engneerng, Unverst of Scence and Technolog Beng Beng, Chna Jun Sun, Satosh Nao Futsu R&D Center Co. Ltd. Beng, Chna Abstract Document mages captured b a moble phone camera often have perspectve dstortons. In ths paper, fast and robust vanshng pont detecton methods for such perspectve documents are presented. Most of prevous methods are ether slow or unstable. Based on robust detecton of tet baselnes and character tlt orentatons, our proposed technolog s fast and robust wth the followng features: (1) quck detecton of vanshng pont canddates b clusterng and votng on the Gaussan sphere space; and () precse and effcent detecton of the fnal vanshng ponts usng a hbrd approach, whch combnes the results from clusterng and proecton analss. The rectfed mage acceptance rate for MobleCam-based documents, sgnboards and posters s more than 98% wth an average speed of about 100ms. Kewords-Vanshng pont detecton, Perspectve document rectfcaton, clusterng, MobleCam-based documents, the Gaussan sphere I. INTRODUCTION Wth wdespread usage of the cheap dgtal camera bultn the moble phone (MobleCam n abbrevaton thereafter) n people s dal lfe, the demand for smple, nstantaneous capture of document mages emerges. Dfferent from the tradtonal scanned mage, lots of the MobleCam-based document mages have perspectve dstortons. Consequentl, rectfng MobleCam-based perspectve document mages becomes an mportant ssue [6][10]. In computer vson, most perspectve correcton methods rel on vanshng pont detecton. And these methods nvolve etractng multple lnes and ther ntersectons, or usng teture and frequenc knowledge [4][11][17][19]. In document analss, there are also varous works on correcton of perspectve documents captured b general dgtal cameras. We dvded the methods of vanshng pont detecton nto two sub groups: drect approaches and ndrect approaches. The drect approaches drectl analze and calculate on mage pels, such as proecton analss from a perspectve vew for horzontal vanshng pont detecton []. These approaches have rather good precsons. But a full or partal search of the space s computatonall epensve, even mpossble. The ndrect approaches convert the orgnal space nto a clue space, and search the vanshng pont n that new small space. Most ndrect approaches nvolve etractng multple straght or llusor lnes and votng vanshng ponts b model fttng [][7][8][9][1][13]. These ndrect approaches are tme effcent. However, the model fttng s senstve. In MobleCam-based document analss, there are two man challenges for rectfng perspectve documents. Frst, the rectfng engne should be hghl precse wth fast speed. The above methods can t cover the two ssues well at the same tme. Second, a MobleCam-based mage s usuall a partal porton of a whole document wth few document boundares. To solve the above problems amng at a practcal MobleCam applcaton, we focus on fast and robust vanshng pont methods for rectfng general perspectve documents. Frst, we propose a hbrd approach for robust real-tme vanshng pont detecton b ntegratng the drect and ndrect approaches effcentl. As for the second challenge, we utlze horzontal tet baselnes and character tlt orentatons as llusor lnes to vote and compute vanshng ponts. The remander of ths paper s organzed as follows. Secton ntroduces the basc prncple of our vanshng pont detecton methods. In Secton 3, we descrbe our vanshng pont detecton process wth a hbrd approach. Secton 4 s about the eperments and result analss. Fnall we conclude the paper n Secton 5. II. BASIC PRINCIPLE OF OUR METHODS The vanshng pont (horzontal or vertcal) n a D space can be descrbed as (v, v. Generall speakng, vanshng pont detecton s to fnd a proper pont accordng to an optmzaton process n the mage plane. That s to sa, ( v, v ) arg ma f (,, (, where f(, s the proft functon for the optmzaton. For the drect approaches for vanshng pont detecton, the search space s. Obvousl, search n such a space s computatonall epensve. We propose a novel and hbrd approach for vanshng pont detecton. Our approach frst votes and clusters lne ntersectons nto vanshng pont canddates (an ndrect approach). Then proecton analss from perspectve vews on these canddates s performed, whch s a drect approach. The vanshng pont s obtaned b an optmzaton of a functon based on the prevous two steps. The functon can be epressed as followng: /11 $ IEEE DOI /ICDAR

2 g(, G( fndrect(,, fdrect(, ), (1) where f ndrect (, and f drect (, are the proft functons for the ndrect and drect approaches respectvel. For vanshng pont detecton, frst, we locate all straght and llusor lnes. Then calculate all ntersectons for ever lne par. These ponts are parttoned nto several groups b clusterng. The cluster center pont s regarded as a tpcal representaton of ts sub regon, whch s C { c c S, 1,..., N}. () Rather than searchng on the whole space n, we search on the representatve pont set n C for speed up. Snce the pont set n C s representatve enough, the searched mamum n C s a good appromaton to the global mamum. Consequentl, the fnal resultng vanshng pont s gven b ( v, v ) argmag(,. (, C Then we perform a drect approach, e.g., proecton analss from a perspectve vew, on the new search space. Compared C n Equaton () wth, the search space of our hbrd approach s ust composed b several ponts, whch s much smaller than that of the drect approaches. Hence, t s tme effcent. If the number of detected lnes s enough, suffcent lne ntersectons wll be generated. And the true vanshng pont wll be embedded n these ntersectons wth a hgh probablt. III. ROBUST VANISHING POINT DETECTION In a horzontal document, each tet row has a clue horzontal drecton. Our method uses document boundares and tet rows to detect the horzontal vanshng pont. However, n the vertcal drecton of a horzontal document, there wll be no tet columns for vertcal clues. Smlar wth the method n [7], we etract the vertcal character strokes as llusor vertcal lnes. We perform robust detecton of tet baselnes and character tlt orentatons b heurstc rules and statstcal analss [15]. A. Clusterng for Vanshng Pont Canddates For vanshng pont detecton n document analss, most researchers drectl voted vanshng ponts wth ntersected lnes or other nformaton on the mage plane. Frstl, we clustered lne ntersectons to vanshng ponts on the mage plane. As we known, on the mage plane, lnes and lne ntersectons are more senstve than the Gaussan sphere when the perspectve dstorton s weak. In such cases, lne ntersecton ponts are dstrbuted n a large (even nearl nfnte) range on the mage. Secondl, we perform vanshng pont canddate clusterng on the Gaussan sphere. (1) Clusterng on the mage plane All horzontal lnes (ncludng straght lnes and smeared lnes detected) are Set( Lne) {( a1, b1, c1),...,( an, bn, cn )}, where N s the number of all horzontal lnes. As we know, two lnes wll produce an ntersecton pont. As a result, there are N P N ( N 1) / ntersectons whch are possble canddates of the horzontal vanshng pont. These ntersectons are Set( Pt) {( 1, 1),..., ( N, )}. P N P After checkng the dstrbuton of lne ntersectons, we dscover that these ntersectons are located n the D space wth one or more groups wth hgh denst. It s natural to partton these ponts nto groups b clusterng. A sample of the dstrbuton of ntersectons for a horzontal vanshng pont s descrbed n Fg. 1. (a) (b) (c) Fg. 1. Intersecton dstrbuton for a horzontal vanshng pont: (a) captured mage, (b) horzontal lnes, (c) pont dstrbuton on the mage plane, and (d) crcle ntersecton dstrbuton on the Gaussan sphere. Our clusterng space s D Eucldean space, and the smlart measure of two ponts s the Eucldean dstance, d(, ) ( ) ( ), where ( ) s the feature vector (a pont n the space). The k-means clusterng algorthm s a rather robust clusterng algorthm, whch s also proved from our ntal eperences. The number of clusters n k-means s specfed b the followng smple rule: Ncluster ma( ln( N P ), 10). () Clusterng on the Gaussan sphere In order to deal wth Hough-based problems and the ssues of vewpont senstvt and mage nose for vanshng pont detecton n perspectve mages [17-19], alternatvel, we propose a clusterng based method for detectng vanshng ponts, whch drectl clusterng proecton ntersectons (crcle ntersectons) from the ntersecton ponts of all pars of lnes n the mage onto the Gaussan sphere. And we use a rough estmaton of the focal length to construct crcle ntersectons on the Gaussan sphere. It s reasonable to cluster these ntersectons nto (d) 137

3 vanshng ponts wth vector representaton on the Gaussan sphere. Wthout loss of generalt, we also descrbe the clusterng based method for locatng the horzontal vanshng pont. Each horzontal lne on the mage plane decdes a great crcle on the Gaussan sphere. And two lnes decde two great crcles. These two crcles have two ntersecton ponts. In fact, onl one-half sphere s used, and we get one ntersecton whch s proected from the lne ntersecton pont on the mage plane [16]. These ntersectons are possble canddates of the horzontal vanshng pont wth vector representaton on the Gaussan sphere. After checkng the dstrbuton of crcle ntersectons, we fnd that these ntersectons are located n the 3D surface wth one or more groups wth hgh denst. It s natural to partton these ponts nto groups b clusterng. A sample of the ntersecton dstrbuton for a horzontal vanshng pont on the mage plane s descrbed n Fg. 1 (c). And the correspondng crcle ntersecton dstrbuton on the Gaussan sphere s shown n Fg. 1 (d). B. Vanshng Pont Detecton and Selecton After clusterng, we wll obtan Ncluster clusters, and the center of each cluster s the average of all the ntersecton ponts n ths cluster. Then, we go back to calculate the vanshng pont canddates on the mage plane from results on the Gaussan sphere. And the followng dscussons are based on the mage plane. Suppose N s the pont number n the th cluster, each canddate has a weght from clusterng whch s gven b w N / c N cluster 1 N. Each weght can be regarded as the proft functon for the ndrect approach,.e., c fndrect (, ) w ( ). Ths weght can be vewed as a condtonal probablt P(( ) Andrect ) fndrect ( ), where Andrect means the vanshng pont s detected b the ndrect approach. In order to get a more stable vanshng pont, we use a drect approach to refne vanshng pont canddates n the above search space [14][15]. For each cluster center, the perspectve proecton analss [] s performed, and the dervatve-squared-sum of the proecton profles from a perspectve vew s calculated, whch s normalzed as a weght value. Smlarl, ths weght can also be seen as a condtonal probablt P(( ) Adrect ) fdrect ( ), where Adrect means the vanshng pont s derved from the drect approach. And the combned probablt of the above two probabltes s 1 P ) f 1 ( ) f (, ). ( ndrect drect And the resultng horzontal vanshng pont s decded b the Baesan crtera ( v, v ) arg ma P(, ). ( ) X C The last step s to confrm the resultng vanshng pont. Our reecton strateg s that the dervatve-squared-sum of the true vanshng pont wll be larger than values of other ponts [14][15]. IV. EXPERIMENTS The eperment database ncludes 418 test samples captured b several moble phone cameras. These mages are n RGB color format wth a resoluton. More than 90% of the mages have perspectve dstortons. Gven a resultng vanshng pont, VP, the relatve dstance from the ground truth VPt s calculated. If VP VPt (3) TVP, VPt then VP s regarded as a correct vanshng pont. In our sstem, the ground truth vanshng ponts are calculated from the manuall marked horzontal and vertcal lnes. When the dfference satsfes Equaton (3) (TVP=1/0), then there s no seemngl perspectve dstorton. We dvde our rectfed mages nto fve groups: (1) HIT, successful for rectfcaton n both horzontal and vertcal drectons; () HHIT, successful n the horzontal drecton; (3) VHIT, successful n the vertcal drecton; (4) REJ, the rectfed mage s the same as the orgnal mage; (5) ERR, error rectfcaton. Fg.. Rectfcaton accurac (%) of each resultng group. Our methods nclude M1 and M. M1 s smlar to M whch are both descrbed n ths paper. The man dfference between them s that M clusters vanshng pont canddates on the mage plane, whle M1 performs vanshng pont canddate clusterng on the Gaussan sphere. We compared our methods to other four methods: M3 doesn t use character vertcal strokes for vertcal vanshng pont detecton; M4 uses the ndrect approach based on clusterng onl to detect vanshng ponts on the mage plane; M5 uses model fttng n [9] nstead of clusterng; and M6 uses a 138

4 sequental correcton stle. The accurac results are descrbed n Fg.. In our dataset, there are even some street sgns and nondocument mages. The fracton of these non-document mages s about 0%. The REJ rates of our methods are 3.68% and 6.3%, whch s manl caused b too large dstortons. For a moble phone wth some proper nteractve GUIs, users ma accept the results of HIT, HHIT, VHIT, and REJ because the resultng mage from these has a much better qualt than (or a same qualt as) the orgnall captured mage. In ths wa, the acceptance rates of our methods are 98.80% and 98.33%. Compared wth M3, our methods (M1 and M) mprove the HIT groups b 16.6% and 11.48% respectvel. Ths shows that character vertcal strokes are ver useful to detect the vertcal vanshng pont for documents wthout vertcal boundares. Compared wth M4, our methods mprove 11.7% and 6.94% for the HIT accurac. Our hbrd approach s more robust for vanshng pont detecton. Compared wth M5, our methods mprove the HIT accurac b 7.17% and.39% and decrease the processng tme b 11ms and 1ms, whch shows that our clusterng strateg s robust and fast compared to the tradtonal model fttng. Our methods have a smlar performance wth M6, but M6 uses a sequental stle wth partal rectfcaton. The processng speed s shown n Table 1, where Tme represents the average processng tme for each mage wthout ncludng the tme for the grascale mage converson and the fnal perspectve transformaton. Eperments are run on a DELL PC wth 3GHz CPU, G Memor on a Wndows XP OS. Table 1. Results of the average processng tme. M1 M M3 M4 M5 M6 Tme (ms) As shown n Table 1, the average processng tme of our methods s largel less than M6, and the reduced tme s more than 100ms. We also test the drect approach b a herarchcal search for horzontal vanshng pont detecton n [], whch s more tme consumng. For one mage n test samples, the detecton tme s more than one second. Compared to M, our new approach (M1) has a hgher accurate rate. The HIT accurac s mproved from 53.11% to 57.89%. Ths shows that vanshng pont canddate clusterng on the Gaussan sphere s effectve. Moreover, the addtonal processng tme s onl 5ms. Our rectfcaton technolog for perspectve document mages has been mplemented and appled nto real moble phones. Real applcatons show that our method has an acceptable performance for both accurac and speed. For a moble phone camera-based document mage (wth a 180*960 resoluton), the average processng tme s about 1s~s. Some real samples and correspondng rectfed mages are shown n Fg. 3. (b) Fg. 3. Some real samples captured b mobles: (a) orgnal mages, (b) rectfed mages. V. CONCLUSIONS Perspectve rectfcaton of MobleCam-based documents faces several challenges, such as speed, robustness, nonboundar documents, etc. In ths paper, we present a fast and robust technolog to deal wth these problems. In our methods, the hbrd approach for vanshng pont detecton combnes drect and ndrect approaches wth hgh precson and fast speed. The eperments on dfferent document mages captured b moble phone cameras show that our method has a good performance wth an average speed of about 100ms on a regular PC. Moreover, our perspectve rectfcaton sstem has been appled nto real moble phones. ACKNOWLEDGMENTS The work of the authors Xu-Cheng Yn and Hong-We Hao s partl supported b the R&D Specal Fund for Publc Welfare Industr (Meteorolog of Chna under Grant No. GYHY and the Fundamental Research Funds for the Central Unverstes under Grant No. FRF-BR B. REFERENCES [1] J.F. Cann, A computatonal approach to edge detecton, IEEE Trans. on PAMI, vol. 8, no. 6, pp , [] P. Clark, and M. Mrmehd, Rectfng perspectve vews of tet n 3D scenes usng vanshng ponts, Pattern Recognton, vol. 36, no. 11, pp , 003. [3] C.R. Dance, Perspectve estmaton for document mages, Proceedngs of SPIE Conference on Document Recognton and Retreval IX, pp , 00. [4] W.L. Hwang, C.-S. Lu, and P.-C. Chung, Shape from teture: estmaton of planar surface orentaton through the rdge surfaces of contnuous Wavelet transform, IEEE Trans. on Image Processng, vol. 7, no. 5, pp , [5] D.X. Le, G.R. Thoma, and H. Wechsler, Automated borders detecton and adaptve segmentaton for bnar document mages, Proceedngs of ICPR, pp , [6] J. Lang, D. Doermann, and H.P. L, Camera-based analss of tet and documents: a surve, Internatonal Journal on Document Analss and Recognton, vol. 7, no. -3, pp , 005. (a) 139

5 [7] S.J. Lu, B.M. Chen, and C.C. Ko, Perspectve rectfcaton of document mages usng fuzz set and morphologcal operatons, Image and Vson Computng, vol. 3, no. 5, pp , 005. [8] C. Monner, V. Ablavsk, S. Holden, and M. Snorrason, Sequental correcton of perspectve warp n camera-based documents, Proceedngs of ICDAR, vol. 1, pp , 005. [9] M. Plu, Etracton of llusor lnear clues n perspectvel skewed documents, Proceedngs of CVPR, pp , 001. [10] S. Pollard, and M. Plu, Buldng cameras for capturng documents, Internatonal Journal on Document Analss and Recognton, vol. 7, no. -3, pp , 005. [11] J. Shufelt, Performance evaluaton and analss of vanshng pont detecton technques, IEEE Trans. on PAMI, vol. 1, no. 3, pp. 8-88, [1] J. Lang, D. DeMenthon, and D. Doerman, Flattenng curved documents n mages, Proceedngs of CVPR, pp , 005. [13] J. Lang, D. DeMenthon, and D. Doerman, Camera-based document mage mosacng, Proceedngs of ICPR, pp , 006. [14] X.-C. Yn, J. Sun, S. Nao, et al., A mult-stage strateg to perspectve rectfcaton for moble phone camera-based document mages, Proceedngs of ICDAR, pp , 007. [15] X.-C. Yn, J. Sun, S. Nao, et al., Perspectve rectfcaton for moble phone camera-based documents usng a hbrd approach to vanshng pont detecton, Proceedngs of Internatonal Workshop on CBDAR, pp , 007. [16] S. Barnard, Interpretng perspectve mages, Artfcal Intellgence, vol. 1, no. 4, pp , [17] R.T. Collns, and R.S. Wess, Vanshng pont calculaton as a statstcal nference on the unt sphere, Proceedngs of ICCV, pp , [18] A. Mnagawa, N. Tagawa, T. Mora, T. Gotoh, Vanshng pont and vanshng lne estmaton wth lne clusterng, IEICE Trans. on Informaton and Sstems, vol. E83-D, no. 7, pp , 000. [19] E. Lutton, H. Matre, and J. Lopez-Krahe, Contrbuton to the determnaton of vanshng ponts usng Hough transform, IEEE Trans. on PAMI, vol. 16, no. 4, pp ,