Global Journal of Computer Sene and Tehnology: F Graphs & vson Volume 7 Issue Verson.0 Year 207 Type: Double Blnd Peer Revewed Internatonal Researh Journal Publsher: Global Journals In. (USA) Onlne ISSN: 0975-472 & Prnt ISSN: 0975-4350 A Vdeo Stablzaton Method based on Inter- Frame Image Mathng Sore By Qnghe Zheng Shandong Unversty Abstrat- Vdeo stablzaton s an mportant vdeo enhanement tehnology whh ams at removng annoyng shaky moton from vdeos. In ths paper, we propose an robust and effent vdeo stablzaton algorthm based on nter-frame mage mathng sore. Frstly, mage mathng s performed by a method ombnng Maxmally Stable Extremal Regons (MSERs) deteton algorthm and Features from Aelerated Segment Test (FAST) orner deteton algorthm, whh an get the mathng sore and the moton parameters of the frame mage. Then, the mathng sore s fltered to flter out the hgh frequeny omponent and keep the low frequeny omponent. And the moton ompensaton s performed on the urrent frame mage aordng to the rato of the mathng sore before and after the flterng to retan the global moton and remove the loal jtter. Varous lassal orner deteton operators and regon mathng operators are ompared n experments. Keywords: vdeo stablzaton, vdeo warpng, moton estmaton, moton ompensaton, partal ompensaton. GJCST-F Classfaton: F.2.2, I.3.3, I.4.0 AVdeoStablzatonMethodbasedonInterFrameImageMathngSore Strtly as per the omplane and regulatons of: 207. Qnghe Zheng. Ths s a researh/revew paper, dstrbuted under the terms of the Creatve Commons Attrbuton- Nonommeral 3.0 Unported Lense http://reatveommons.org/lenses/by-n/3.0/), permttng all non-ommeral use, dstrbuton, and reproduton nany medum, provded the orgnal work s properly ted.
A Vdeo Stablzaton Method based on Inter- Frame Image Mathng Sore Qnghe Zheng Abstrat- Vdeo stablzaton s an mportant vdeo enhanement tehnology whh ams at removng annoyng shaky moton from vdeos. In ths paper, we propose an robust and effent vdeo stablzaton algorthm based on nter-frame mage mathng sore. Frstly, mage mathng s performed by a method ombnng Maxmally Stable Extremal Regons (MSERs) deteton algorthm and Features from Aelerated Segment Test (FAST) orner deteton algorthm, whh an get the mathng sore and the moton parameters of the frame mage. Then, the mathng sore s fltered to flter out the hgh frequeny omponent and keep the low frequeny omponent. And the moton ompensaton s performed on the urrent frame mage aordng to the rato of the mathng sore before and after the flterng to retan the global moton and remove the loal jtter. Varous lassal orner deteton operators and regon mathng operators are ompared n experments. And expermental results llustrate that the proposed method s effetve to stablze translatonal, rotatonal, and zoomng jtter and robust to loal motons, and has the state-of-the-art proessng speed to meet the needs of real-tme equpment. Keywords: vdeo stablzaton, vdeo warpng, moton estmaton, moton ompensaton, partal ompensaton. I. Introduton Vdeo enhanement s gettng more and more attenton wth the nreasng popularty of dgtal vsual meda. As one of the most mportant ways of enhanement, vdeo stablzaton s a tehnque for removng abnormal mage offsets suh as jtter and rotaton, et., by dgtal mage proessng. One of the most obvous dfferenes between professonal and amateur level vdeo s the qualty of amera moton; hand-held amateur vdeo s typally shaky and undreted whle professonals use areful plannng. Gven the unstable vdeo, the vdeo stablzaton s desgned to synthesze new mage sequenes seen from the new stable amera trajetory. The typally algorthm manly onssts of the followng parts: feature pont extraton, feature pont mathng, moton estmaton, moton ompensaton, synthess of new vdeo sequenes. Pror tehnques for software vdeo stablzaton follow two man approahes, provdng ether hgh qualty or robustness and effeny. At present, the most ommonly used vdeo mage stablzaton method s 2D stablzaton [], whh s wdely used n ommeral software and mltary. Ths method s sutable for the 2D moton models, whh s very effetve for the affne or projeton transformaton of the urrent frame. However, due to the nablty to smulate the amera movement aused by the dsparty and other ssues, the two-dmensonal moton model s very fragle and poor stablty. Then, 3D vdeo stablzaton tehnque was proposed by Buehler n 200 [2] and developed by Lu n 2009 [3], whh shows a strong stablty and has the ablty to smulate the ameras 3D trajetory. In ths method, a new struturefrom-moton (SFM) tehnque [4] s used to onstrut the 3D model of the bakground and amera moton, and then varous new flterng deas are started around the new 3D trajetory model [5, 6]. But SFM s a fundamentally dffult problem, and the generalty of urrent solutons s lmted when appled to the dverse amera motons of amateur-level vdeo. The problem wth 3D stablzaton and 2D stablzaton s opposte: the 3D model s too omplex to be alulated n real tme and the robustness s too poor. So t s dffult to use the 3D mage stablzaton tehnology n daly busness and medal treatment. In general, requrng 3D reonstruton hnders the pratalty of the 3D stablzaton ppelne. In ths paper, we ntrodue a robust and effent method for software vdeo stablzaton. In spte of the mage stablzaton platform has been wdely used n professonal equpment and aheved good results, t stll requres addtonal hardware support, and snt sutable for amateur onsumers. For example, vdeo qualty wll be severely redued due to amera vbraton n stuatons lke takng ptures by a tourst enthusast on a bumpy ar. Year 207 F Author: Shool of Shool of nformaton sene and Engneerng Shandong Unversty Jnan, Shandong, P.R. Chna. e-mal: 7896307@qq.om 35 Global Journal of Computer Sene and Tehnology ( ) Volume XVII Issue I Verson I 207 Global Journals In. (US)
F A Vdeo Stablzaton Method based on Inter- Frame Image Mathng Sore Feature pont mathng Edge mathng Feature regon mathng Vdeo Sequene Image pre-proessng (seletable) Image mathng Moton estmaton Moton ompensaton Image enhanement Image denosng Image orreton Year 207 36 Global Journal of Computer Sene and Tehnology ( ) Volume XVII Issue I Verson I II. Vdeo Stablzaton Algorthm Vdeo stablzaton manly nludes four stages: mage pre-proessng, mage mathng, moton estmaton and moton ompensaton (see Fg. ). Image pre-proessng s to elmnate the nterferene of fuzzy, gray shft and geometr dstorton aused by the nonssteny of the lght n the proess of obtanng the vdeo, whh s able to redue the dffulty of mage mathng and mprove the auray of mage mathng. Image mathng s the key step of vdeo stablzaton, whh dretly determnes the qualty of the fnal vdeo. The purpose of mage mathng s to fnd a spatal transformaton, so that the oordnates of the overlappng parts n the mage an be aurately mathed. Image mathng algorthm needs not only to ensure the auray of mage mathng, but also to mnmze the amount of omputaton. Moton estmaton s a omplete set of tehnques for extratng moton nformaton from vdeo sequenes. The man ontent of moton estmaton s how to get enough moton vetors qukly and effetvely aordng to the oordnates of mathng feature ponts. Moton ompensaton s to predt and ompensate the urrent mage by the prevous mage, and to ompensate the orrespondng moton nformaton of the prevous frame aordng to the moton vetor. The key of moton ompensaton s to dstngush loal jtter and global moton effetvely, whh makes the fnal vdeo get a good vsual effet. III. Image Mathng Vdeo Sequene Output In ths part, we wll ntrodue two lassal mage mathng algorthms whh are used n the fourth part: MSERs algorthm [7] proposed n 204 and FAST orner deteton algorthm [8] proposed n 202 used for vdeo stablzaton. a) Regon-based mathng algorthm MSERs use the onept of a watershed n the terran to fnd a stable loal area. Prevous watershed transforms were manly used for mage segmentaton. The algorthm foused on the water level at the tme of regonal mergng. At ths tme, the small water puddles and ponds were unstable and the onneted water Fgure : Vdeo frame stablzaton algorthm flow hart volume hanged drastally. Strtly defned from the mathematal pont of vew, MSER s a regon whh has the smallest hange n the number of pxels at a gven threshold. MSERs s urrently reognzed as the best performane of the affne nvarant regon. Algorthm Steps The pxels of a gven mage are ordered n gray sale values. Add the pxels nto mages n aordane wth asendng or desendng and lnk the area. Defne Q as an arbtrary onneted regon n the bnary mage orrespondng to the threshold value. When threshold hanges n (-, + ), onneted regons orrespondng to Q + and Q -. Wthn ths range of varaton, the regon q() wth mnmal hange rate s onsdered to be MSERs. b) Feature-based mathng algorthm FAST s a orner deteton method, whh an be used for the extraton of feature ponts and the ompleton of trakng and mappng objets. The most promnent advantage of ths algorthm s ts omputatonal effeny and good repeatablty. The bas prnple of the algorthm s to use a rumferene of 6 pxels (a rle wth a radus of 3 pxels drawn by the Bresenham algorthm) to determne whether the enter pxel P s the orner pont. Then the enter pxel s alled the orner pont: If the brghtness of N pxels on the rumferene are larger than the sum of enter pxel and a threshold T, or smaller than the dfferene between the enter and the threshold T. In an mage, the non orner ponts are more easly measured and aounted for the majorty of the pxels. Therefore, the frst elmnaton of non orner ponts wll greatly mprove the deteton rate of orner ponts. Algorthm steps Detet the non orner ponts on the rle. Determne whether the enter pont s a orner pont and make a orner deteton for eah pont on the rle f t s true. Remove the non-maxmum orner and get the output orner pont. 207 Global Journa ls In. (US)
Year 207 F A Vdeo Stablzaton Method based on Inter- Frame Image Mathng Sore Calulate the sore funton and ompare t n the negh borhood of 3*3. IV. Moton Estmaton 2D parametr moton model s used for the moton of the amera (see Fg. 2). The movng amera s attahed to the oordnate system O-XYZ and the orrespondng projeton onto the mage plane s attahed to the system O-PQ. The amera moton onssts of two omponents: a translaton (T x, T y, T z ) T and a rotaton (α, β, γ) T, whh represent roll, pth and yaw of the moton. A pont wth an mage oordnate (p, q) n the spae (x, y, z) wll move to another loaton (x, y, z) wth an mage oordnate (p, q) and the foal length f wll beome f through nter frame moton. The relatonshp of orner ponts n spae and the mage plane s defned by Eq. () and Eq. (2), respetvely. a, b,, d, e, f, g, h, among the equaton s the parameters of moton matrx. 37 Image plane and the oordnate plane x a y d z g p f q f b e h ap + bq + f gp + hq + f dp + eq + ff gp + hq + f x Tx f y Ty z T z ftx ftz f T f Tz If the rotaton angle of the olleted vdeo sequene frame mage n the amera moton proess s less than 5, Eq. (2) an be approxmated as: Then Eq. (2) an be expressed as: Fgure 2: p + αqγf ftx p f γp βq + f ftz αp + q + βf fty q f γp βq + f ftz Let ( ) Two equatons are provded by eah set of mathng orner ponts, thus 2N equatons wll be provded by N pars, and subsequently, the moton parameters an be obtaned by the least square soluton. a) Feature pont seleton In the tradtonal method, the moton equaton s obtaned by detetng and mathng the feature ponts between the frames. Sne there are a large number of mathng feature ponts n two adjaent frames to solve s p βq + f ftz / f y γ (4) s p p + αq γf ftx s q αp + q + βf fty a moton equaton ontanng only four parameters, there s a large omputatonal redundany. At the same tme, mage feature pont mathng s prone to msmath. So the tradtonal methods need to add a wld pont elmnaton funton, whh used to remove unrelable feature ponts that easly lead to false mathes. We propose a novel feature pont deteton method for solvng the equatons of moton ombnng the advantages of feature pont deteton method and regon deteton method. Frstly, MSERs deteton s performed on eah mage n the vdeo sequene (see () (2) (3) (5) Global Journal of Computer Sene and Tehnology ( ) Volume XVII Issue I Verson I 207 Global Journals In. (US)
F A Vdeo Stablzaton Method based on Inter- Frame Image Mathng Sore Fg 3.a ). The seond step uses the retangle to label the ellptal areas whh have a stablty of the top three. The thrd step s the FAST feature ponts deteton n the marked area (see Fg 3. b ). Year 207 38 Global Journal of Computer Sene and Tehnology ( ) Volume XVII Issue I Verson I (a) Fgure 3: The MSERs area marked by retangleand FAST orner pont deteton b) Feature pont seleton To trak feature ponts, a wndow P*P entered at eah seleted pont s desgned and mathed usng damond searh (DS) method and the sum of absolute dfferene rteron (SAD) [9]. The searhng area s (P+2M) (P+2N), where M and N are maxmum horzontal and vertal dsplaements, respetvely. Thus, the orrespondng pont s at the enter of the mathng wndow. Moreover, two ssues are onsdered n dedng the proper sze of the feature wndow: A large sze would ause a dsloaton of pxels, but a small sze offers less nformaton. In pratal use, a feature wndow wth a sze 9*9 has a good performane expermentally. Next, the frst hapter of the N ptures n eah seond of the vdeo sequene s set as a referene frame, and the remanng mages n eah seond are mathed wth the referene frame for feature ponts. Fnally, the least squares method s used to solve the moton equaton through the oordnates of mathng feature ponts. In ths way, stable and effetve feature ponts an be obtaned, whh s more robust to nose suh as llumnaton. At the same tme, the reduton of the number of feature ponts an solve the equatons of moton more qukly. ) Computng moton parameters The Eq. (6) ndates that the moton nludes four parameters: the rotaton Δθ, the translaton (Δx, Δy), and the salng Δλ. Gven a set of N mathed pars, Δλ an be defned as: d d N d d λ (6) N d d where, ( u U ) 2 + ( v V ) 2 (7) 2 ( u U ) + ( v V ) 2 (8) (b) where (U, V) represents the bary enter of the ponts n the urrent frame, and (U, V ) represents the bary enter n the referene, respetvely. Then applyng Eq. (6) to a set of N pars of mathng feature ponts, 2N lnear funtons an be obtaned wth three unknowns m[δθ, Δx, Δy] T. The fnal funton BAm s n the form of a matrx, as shown n Eq. (9). u λu λv v λv λu B..., A... un λun λvn vn λvn λun 0... 0 0... 0 To get the moton parameters, the ntal solutons are obtaned by pseudo nverset transformaton and then refned by Levenberg-Marquardt (LM) method [0]. Frstly, n(n 2) pars of ponts wth mnmal SAD n the template mathng proessng are seleted and the ntal value of m an be omputed by m (A T A) - A T B. Then, the L-M method s used to refne solutons by mnmzng the square of oordnate dfferenes. Let (u, v ) T and (U, V ) T denote the known feature ponts and the estmated ponts, respetvely. The objet funton s defned as Eq. (0). N T 2 2 u U u U T [( u U ) + ( v V ) ] e e E v V v V V. Moton Compensaton (9) (0) At ths stage, t s lear that only the unwanted amera jtter should be removed n the moton of the amera. We suppose that the moton of the amera s usually smooth wth slow varaton and unwanted amera jtter nvolves rapd varaton. From another pont of vew, the hgh frequeny omponent of the moton vetor s onsdered to be an unwanted amera jtter and an be fltered out by a low pass flter. On the bass of ths dea, we propose a partal bakward ompensaton method wth a novel flterng algorthm. Frstly, the mathng sores G of -th mage and 207 Global Journa ls In. (US)
F A Vdeo Stablzaton Method based on Inter- Frame Image Mathng Sore referene mage n the vdeo sequene are defned as: the number of suessful mathng orner ponts dvded by the total number of orners. The hgher the approxmaton degree between the referene frame and the urrent frame, the hgher the mathng sore. On the next step, one dmensonal dsrete wavelet transform (DWT) s used to remove the hgh frequeny nformaton, and then one dmensonal dsrete wavelet reonstruton (IDWT) s performed. The Haar wavelet s used to arry out the wavelet transform. Wavelet transform s the nhertane and development of tradtonal Fourer transform. Beause the multresoluton analyss of wavelet has good loalzaton property n spatal doman and frequeny doman, t an be used to analyze the arbtrary detals of the objet gradually. Retanng the low-frequeny nformaton, whh s equvalent to retanng the global movement and removng the jtter. Fnally, adopt the partal ompensaton prnple: the moton parameters are ompensated aordng to the rato of the mathng sores before and after the flterng, whh s gven by Eq. (), where G b and G a s the mage mathng sore before and after the wavelet transform. A strategy wll be used n the mage mathng sore alulaton phase: remove the urrent frame wth mathng sores below a preset threshold N (30 n ths paper). In the end, the proessed mage frame s made nto a new vdeo. x y G θ G λ b a x y θ λ () Compared to the tradtonal algorthms, whh need to flter the horzontal dsplaement, the vertal dsplaement and the rotaton angle, our method only needs to flter the mage mathng sore, whh mproves the effeny and meet the real-tme requrement. At the same tme, the partal ompensaton method based on the mage mathng sore an better retan the global moton and avod the phenomenon of over smoothng. Table : The performane of varous lassal orner deteton operators and regon deteton operators Method Mean value of orners Computatonal tme(s) SIFT [] SURF [2] Harrs [3] FAST MSERs+SIFT MSERs+SURF MSERs+Harrs MSERs+FAST 8 65 35 59 2 47 34 9 299.62 5.96 96.68 36.4 46.88 60.29 29.8 2.82 Fgure 4: Comparson of PSNR of several lassal operators Year 207 39 Global Journal of Computer Sene and Tehnology ( ) Volume XVII Issue I Verson I Fgure 5: The mean mage of 0 onseutve frame mage 207 Global Journals In. (US)
Year 207 Global Journal of Computer Sene and Tehnology ( ) Volume XVII Issue I Verson I F A Vdeo Stablzaton Method based on Inter- Frame Image Mathng Sore 40 VI. Expermental Results Ths part presents expermental results obtaned from a vdeo sequene, whh s wdely used by varous vdeo proessng laboratores. The experments are arred out on MATLAB R203a wth a 5-4460 CPU. The nput vdeo has a resoluton of 360 240 and nludes 400 frame mages n 0s.Frstly, we ompare the performane of varous lassal orner deteton operators and regon deteton operators(see Tab.). The experment results show that method ombnng MSERs and FAST has a faster omputng speed than the tradton algorthm by fndng and mathng the feature ponts of the entre mage. To make an objetve evaluaton of the vdeo stablzaton algorthms, the peak sgnal-to-nose rato (PSNR) an be used as a measure. In Fg. 4, we ompare the PSNR of 40 mean mages proessed by a varety of operators and tradtonal algorthm. We an see that the performane of the new algorthm s generally better than the tradtonal algorthm from the experment results. To make a subjetve evaluaton of the results, the mean mage of frst 0 onseutve frame mages n the orgnal and stablzed vdeo sequenes are gven respetvely, as shown n Fg. 5. VII. Conlusons A robust and fast vdeo stablzaton method s proposed, whh onssts of mage mathng based on MSERs deteton and FAST orner deteton, moton estmaton and moton ompensaton based on nterframe mathng sore. The partal ompensaton method based on nter-frame mathng sore effently removes flutuatons and retans global moton. The speed optmzaton of algorthm and ts low ost and low requrements of equpment hardware makes t possble to be used for non-professonal amera enthusasts and the portable eletron equpment lke hand-held vsual ommunaton deve. The most tme-onsumng phase of the algorthm s the area deteton. A more smple and effetve feature regon deteton method and fast sortng algorthm an make t faster, whh needs to be further optmzed n future researh. VIII. Aknowledgment Ths work s partally supported by Shandong Provnal Natural Sene Foundaton, Chna (ZR204 FM030 and ZR204FM00). Referenes Référenes Referenas. Ratakonda K. Real-tme dgtal vdeo stablzaton for mult-meda applatons[c]. IEEE Internatonal Symposum on Cruts and Systems. 998:69-72 vol.4. 2. Buehler, C., Bosse, M., & Mmllan, L. (200). Non- Metr Image-Based Renderng for Vdeo Stablzaton. Computer Vson and Pattern Reognton, 200. CVPR 200. Proeedngs of the 200 IEEE Computer Soety Conferene on (Vol.2, pp.609). 3. Lu, F., Gleher, M., Jn, H., & Agarwala, A. (2009). Content-preservng warps for 3d vdeo stablzaton. Am Transatons on Graphs, 28(3), 34-352. 4. Hartley, R., Zsserman, A.: Multple vew geometry n omputer vson. Cambrdge Unversty Press 30(9-0), 865 872 (2006). 5. Matsushta, Y., Ofek, E., Ge, W., Tang, X., & Shum, H. Y. (2006). Full-frame vdeo stablzaton wth moton npantng. IEEE Transatons on Pattern Analyss & Mahne Intellgene, 28(7), 50-63. 6. Lu, F., Gleher, M., Wang, J., Jn, H., & Agarwala, A. (20). Subspae vdeo stablzaton. Am Transatons on Graphs, 30(), 623-636. 7. Bswas P K. Improvng Vdeo Stablzaton Usng Mult-Resoluton MSER Features[J]. Iete Journal of Researh, 204, 60(5):373-380. 8. Xu J, Chang H W, Yang S, et al. Fast feature-based vdeo stablzaton wthout aumulatve global moton estmaton [J]. IEEE Transatons on Consumer Eletrons, 202, 58(3):993-999. 9. Touraps A M, Au O C L. Predtve moton vetor feld adaptve searh tehnque (PMVFAST): enhanng blok-based moton estmaton [J]. Proeedngs of SPIE - The Internatonal Soety for Optal Engneerng, 200, 430:883--892. 0. Marquardt D W. An Algorthm for Least-Squares Estmaton of Nonlnear Parameters [J]. Journal of the Soety for Industral & Appled Mathemats, 2006, (2):43-44.. Zhang Y, Yao H, Xu P, et al. Vdeo stablzaton based on saleny drven SIFT mathng and dsrmnatve RANSAC[C]. Ims 20, the Thrd Internatonal Conferene on Internet Multmeda Computng and Serve, Chengdu, Chna, August. 20:65-69. 2. Huang K Y, Tsa Y M, Tsa C C, et al. Vdeo stablzaton for vehular applatons usng SURFlke desrptor and KD-tree[C]. Internatonal Conferene on Image Proessng. 200:357-3520. 3. Huang K Y, Tsa Y M, Tsa C C, et al. Feature-based vdeo stablzaton for vehular applatons[c]. IEEE Internatonal Symposum on Consumer Eletrons. IEEE, 200:-2. 207 Global Journa ls In. (US)