A New Method for Automatically Labeling Aircrafts in Airport Video

Transcription

1 MAEC Web of Conferences, ( 015) DOI: / matecconf/ C Owned by te autors, publsed by EDP Scences, 015 A New Metod for Automatcally Labelng Arcrafts n Arport Vdeo Xao Luo, Yong ang*, Honggang Wu & Dongln He. e Second Researc Insttute of CAAC, Cengdu, Scuan, Cna ABSRAC: For te problem tat te arport vdeo montorng could only provde te mage nformaton wle te label nformaton ncludng te flgt number s not provded, a new metod for automatcally labelng arcrafts n arport vdeo troug te fuson of vdeo and ADS-B data as been proposed. Frst, te mage coordnates of arcrafts wll be obtaned troug te mage trackng of vdeo. en, te omograpy matrx between two projecton planes wll be calculated wt te four and above pont and lne correspondences selected from te arport map and vdeo mage, respectvely to te map mage coordnates nto te map coordnates. Fnally, te arcrafts n vdeo can be automatcally labeled troug te fuson of mage trackng data and ADS-B montorng data. Because an mage coordnate measurement error exsts at te tme of selectng ponts from te mage, te resultng coordnate converson error s derved and te mpact of pont correspondence geometrc layout on mes coordnate mappng error s analyzed. Experments ave been conducted based on te actual data of Cengdu Suanglu Internatonal Arport. e results sow tat te metod can automatcally label arcrafts n vdeo n an effectve way. Keywords: automatcally labelng; vdeo montorng; ADS-B; data fuson; Homograpy Matrx; error analyss 1 INRODUCION e movement of arcrafts s montored troug nstallng montorng cameras n mportant areas suc as runway, taxway, gate poston, and so n to elp controllers to control te arport ground traffc n an effectve way. Some large arports ave a number of runways and te ground layout s complcated. ere are often some dead angles nvsble by vsual nspecton n te control tower. en, te camera becomes an mportant means for controllers to ntutvely grasp te ground movements. As a non-cooperatve sensor, te montorng vdeo only provdes te targets mage nformaton and cannot provde te label nformaton of targets. Controllers are also requred to determne te label nformaton of targets n vdeo troug vewng te flgt plan, AC montorng screen, and so on. As a cooperatve sensor, ADS-B (Automatc Dependent Survellance-Broadcast) can provde target locaton, speed and oter postonng nformaton as well as te label nformaton ncludng te flgt number. In ts paper, te label nformaton of ADS-B s added nto te vdeo montorng troug data fuson to aceve automatc labelng of te vdeo trackng targets. SYSEM SCHEME e overall system sceme s as sown n Fgure 1. Frstly, a frame of mage s extracted from te vdeo and ten feature ponts are extracted from te mage to fnd out te obvous pont or lne marks on te arport *Correspondng autor: tangyong1979@16.com ground. Meanwle, te accurate map coordnates of tese pont or lne correspondences wll be found out on te arport map. e omograpy matrx between te vdeo mage and arport map can be calculated wt over four pars of correspondences. Background dfferencng [1,] and KL algortm are appled to te arport vdeo to aceve detecton and trackng of mage targets, and te center of mage target trackng s taken as te poston coordnates of te arcraft mage. e omograpy matrx s utlzed to transform te map coordnates of arcraft n ADS-B montorng data nto te mage coordnates n vdeo. Fnally, te target poston data n ADS-B and vdeo trackng data are subject to te data fuson. e K-nearest negbor algortm and oters are appled to aceve te data assocaton [8], tus correlatng te label nformaton ncludng te flgt number n ADS-B to te vdeo to aceve te automatc labelng. 3 CAMERA CORRECION Camera correcton s for acevng nter-converson of te mage coordnates of vdeo trackng targets and te actual arport map coordnates and s a crtcal step for acevng automatc labelng n vdeo. If te arport map s a projectve plane and te vdeo mage s anoter projectve plane, te projecton transformaton relatonsp between bot planes can be descrbed wt a 3 3omograpy matrx H [9-11], as sown n Fgure. A par of ponts X X on te arport map and vdeo mage as te followng coordnate transformaton relatonsp: s s an Open Access artcle dstrbuted under te terms of te Creatve Commons Attrbuton Lcense 4.0, wc permts unrestrcted use, dstrbuton, and reproducton n any medum, provded te orgnal work s properly cted. Artcle avalable at ttp:// or ttp://dx.do.org/ /matecconf/

2 MAEC Web of Conferences A frame of mage Arport map Camera Feature extracton Feature extracton arget detecton and trackng Homograp c calculaton Coordnat e mappng Data fuson Automatc labelng ADS-B Poston coordnat es Fgure 1. Overall System Sceme X HX (1) Projecton Center Vdeo mage plane xy ( xy) v 0 (4) e way to wrte equatons (3) and (4) nto a matrx form s as follows: A 0 (5) Fgure. X o put te map coordnates ( xy, ) and mage coordnates ( uv, ) nto te tree-dmensonal omogeneous column vectors X ( x, y,1) and X ( u, v,1), te equaton (1) can be wrtten as follows: u x 1 3x v H y y () Wat s sgnfcant to omograpy matrx H s te rato of matrx elements. Among te nne elements of H, tere are 8 ndependent ratos. erefore, omograpy matrx H as egt degrees of freedom. In te equaton, omograpy matrx H multpled by a nonzero scale factor wll not cange te projecton transformaton relatonsp, tat s, omograpy matrx can only be determned wt te dfference of only one nonzero scale factor. wo lnear equatons n relaton to te elements of H are obtaned after removng te n equaton () [1]: x y ( xy) u 0 (3) X Scematc Dagram of Homograp Arport ground plane Weren: x y ux uy u A x y 1 vx vy v ( ). It means tat te correspondence of eac par of ponts can provde two ndependent lnear equatons. en, four pars of ponts sould provde egt ndependent lnear equatons. Four 9 matrxes A can be superposed to consttute a 8 9 matrx A. e sole lmt s tat any tree ponts out of four pont correspondences are non-collnear. As te rank of A s egt and te equaton set A 0, tere s a one-dmensonal null space, tus obtanng te determned soluton wt te dfference of only one scale factor. Once te omograpy matrx H s obtaned, te nter-converson between te mage coordnates and map coordnates of any pont can be aceved troug equaton (). Wen tere are over four groups of pont correspondence, A 0 s an over-determned equaton set. If te measured values of all pont correspondences are accurate, te rank of A s stll egt and A as a one-dmensonal null space, for wc exact soluton of can be obtaned. However, n fact, tere s error n te coordnate measurements of pont correspondence due to te mpact of nose, resultng n no accurate soluton of te over-determned equaton A 0. en, we can only defne a cost functon to obtan an approxmate soluton to let te cost functon to ave te mnmum value. Generally, algebrac dstance A s taken as te cost functon. In order to ensure tat s not te vector 0, a norm condton 1 s added. e algortm obtaned s called drect lnear transformaton ( DL ) algortm. ; p.

3 ICEA 015 Sometmes, t s dffcult to select suffcent pont correspondences on te arport map and vdeo mage. In addton to selecton of pont correspondence to calculate omograpy matrx, lne correspondence can also be used to calculate omograpy matrx; te experments n te lterature by Wang[13] ndcate tat te omograpy matrx estmaton metod based on lne correspondence as ger precson and better robustness tan te pont correspondence estmaton. Accordng to te dualty prncple of two-dmensonal projectve geometry, te teorem applcable to ponts s also applcable to lnes, and ponts and lnes are ntercangeable. erefore, n addton to pont correspondence, te lne correspondence can also be used to calculate omograpy matrx. e lnear equaton on plane s axbyc 0, weren a, b and c are parameters of stragt lne. erefore, t s te same wt te omogeneous vector representaton of ponts tat stragt lnes can also be represented as omogeneous vectors ( abc,, ). Assumng a par of lne correspondence l, l on arport map plane and vdeo mage, ten te pont X on te stragt lne l and te pont X on te stragt lne l can be respectvely expressed as follows: l X =0 (6) l X =0 (7) It can be seen from equaton (1) tat X=HX. e followng equaton can be obtaned after puttng X=HX n equaton (7): l HX=0 (8) roug comparng equatons (7) and (8), te followng equaton can be obtaned: l=h l (9) Assumng l ( x, y,1) and l ( u, v,1), ten: x u u y H v v 1 1 (10) e matrx as sown below can also be obtaned after removng te n te equaton (10): u 0 ux v 0 vx 1 0 x Aj 0 u uy 0 v vy 0 1 y Assumng tat tere are n pont correspondences and m lne correspondences, a (n m) 9 matrx A can be obtaned. e optmal soluton can be obtaned from te over-determned equaton set A 0., 4 COORDINAE CONVERSION ERROR e key for realzng arcraft automatc labelng n te vdeo s to make te mage coordnates of target and ADS-B survey coordnates realze mutual converson possbly. e accuracy of coordnate converson wll drectly nfluence te effects of automatc labelng. In ts secton, we wll dscuss te converson error between te mage coordnates and map coordnates. e coordnates correspondng to pont ave measurement error, wc results n tat calculated omograpy matrx generally wll not map object X pont X on te arport map as te mage pont n te mage accurately, tat s to say, error d( X, X ) X exsts between te calculated mage pont and real mage pont X. Assumng tat correspondng measurement on te arport map s accurate, ts can be ensured by accuracy of arport map. We only dscuss te coordnate converson error due to correspondng coordnate measurement error of te ponts selected n te mage. For unrelablty of omograp, t can be measured by covarance matrx of omograpy matrx H. Map X( x, y, w) HX, and wrte down X ( x, y ) ( x/ w, y/ w), X s derved as below: x X / X 3 w y ; H s derved as below: 1 X 0 xx X / w 0 X xx xx, were s nne-dmensonal column vector composed by elements of H, Wrte down te be obtaned tat 1 3 j j t row of H as., and t can Coordnate x and coordnate y of te nt matcng pont X ( 1,, n ) n te mage make up n -dmensonal measurement vector X. e process for calculatng an estmatve converted covarance matrx s as followng [9, 10]: (1) Estmate converted Ĥ accordng to data of marcng pont and elements of Ĥ consttutes nne-dmensonal vector ĥ ; () Calculate te value of Jacoban matrx J X / at ĥ, and te calculaton formula s gven n (b); Wrte Jacoban matrx n te form of parttonng, namely J ( J 1, J, J,,, Jn ), and te followng equaton can be obtaned: p.3

4 MAEC Web of Conferences 1 X 0 xx J X / w 0 X yx, were X nd- cates vector ( x, y,1) (3) Estmate covarance matrx ( J 1 X J) of Assumng tat measurement of X s error-free and measurement of X s mutually ndependent, namely, X s an ndependent gaussan random varable. If tere s a standard devaton of m pxels at eac coordnate drecton, tat s, covarance matrx and ten map world coordnate of blue pont on te cessboard to te mage. Repeat for 300 tmes and draw 3 ellpse dstrbuton stuaton of subpont. It can be seen tat from Fgure 4, projecton scatterng ponts are mostly located wtn two-dmensonal of X s m 0 X 0 m me, accordngly,, terento, 1 ( J X J) m ( JJ) J X/. Once covarance matrx of s determned, te error of mage pont X mapped n te mage by any object pont X on te map can be calculated, and covarance matrx of X s as below: xx xy X JJ xy yy Fgure 3. Cessboard Poto Wat we generally concern about s te dstance from te mage target calculated by omograpy matrx and te real mage target, tat s, root-mean-square error at poston X : RMSE trace( ) xx yy X It can be proved tat [9,10], for dentty mappng, f te ponts selected are unformly dstrbuted on a unt crcle correspondngly, ten RMSE 1 ( x y ) 1r 4 at s to say, dstrbuton of X depends on radal dstance of X. e farter dstance from te target to te center of crcle s, te larger te error s. Below we wll use a ceckerboard poto for test and carry out smulaton verfcaton for te above error calculaton equaton. Fgure 3. s a ceckerboard poto, were te red ponts numbered as 1-6 n te fgure are correspondng to te ponts selected for calculatng omograpy matrx, and te blue ponts are te test ponts used for calculatng coordnate converson result. Assumng tat correspondng world coordnates of te ponts on te ceckerboard ave no error, measurement error of mage coordnate comples wt te Gaussan dstrbuton wt mean value of 0-pxel and standard devaton of tree pxels. For pont correspondence n te mage, gaussan nose generated every tme tat comples wt mean value of 0 and standard devaton of tree s added to coordnates correspondng to pont to calculate H, Fgure 4. 3 Ellpse and Scatterng Ponts Perfectly Identcal roug Monte Carlo Smulaton Gaussan dstrbuton 3 Experment for 300 mes ellpse, wc ndcates tat te error calculaton formula s relable. A projecton of test pont coordnate projecton error fnally obtaned by respectvely selectng two sets of ponts numbered as 1,,3,4 and 1,,3,5 to correspondngly calculate omograpy matrx s sown n Fgure 5. It can be seen tat, four ponts are used to correspondngly calculate omograpy matrx, but dfferent geometrc dstrbuton causes dfferent projecton error, wc ndcates tat correspondng geometrc dstrbuton of pont as mportant nfluence on projecton error, and omograpc pont sall try to surround te area needng coordnate converson f we want to reduce coordnate converson error. e error compar p.4

5 ICEA 015 son between four and sx pont correspondences s sown n Fgure 6, were te blue color ndcates correspondng 3 ellpse of four ponts, and te red color ndcates correspondng 3 ellpse of sx ponts It sows tat, te more pont correspondences selected s, te smaller te error s. An error dstrbuton curve of omograpy matrx correspondngly calculated out by te ponts numbered as 1,,3,4 s sown n Fgure 7. And Fgure 8 s an error dstrbuton curve of omograpy matrx correspondngly calculated out by te ponts numbered as 1,,3,5. It can be seen from Fgure 7 and Fgure 8 tat, Fgure 5. 3 Error Ellpse of Four Homograpc Ponts Fgure 6. 3 Error Ellpse of Four and Sx Homograpc Ponts Fgure 8. Error Dstrbuton Curves Correspondng to Ponts 1,,3,5. error dstrbuton curve canges wt correspondng geometrc dstrbuton of pont, and te closer dstance to pont correspondence s, te smaller te error s. 5 EXPERIMENAL RESULS A Prosllca GC1350C ndustral camera wt te resoluton of produced by German AV Company was used for te ste vdeo recordng n te control tower of Cengdu Suanglu Internatonal Arport on June 15, 011. e ADS-B montorng data Fgure 7. Error Dstrbuton Curves Correspondng to Ponts 1,,3,4 Fgure 9. Error Dstrbuton Curve of Arport Coordnate Converson p.5

6 MAEC Web of Conferences tat s syncronous wt te vdeo s from te ADS-B ground staton developed by e Second Researc Insttute of CAAC nstalled n te arport. Fgure 9 s a curve to calculate omograpy matrx and analyze error dstrbuton by respectvely selectng fve relatvely obvous mark ponts from te mage and te arport map accordng to te ntercepted one frame of vdeo mage. For pont correspondence on te mage and te map, manual selecton s used and t s assumed tat pont correspondence coordnates on te map are accurate and pont correspondence mage error on te mage comples wt Gaussan dstrbuton wt fve-pxel standard devaton. It can be seen from te fgure tat, te arport apron area surrounded by fve ponts as relatvely small error, te error of runway and tax-way at te top left of te mage s obvously ncreased because tey are relatvely far away from te pont correspondence dstrbuton area. Homograp s a projecton converson relatonsp between two planes, terefore, coordnate converson between surface area n te mage and te map can be realzed only, and te ar area n te mage asn t been n te arport plane, so suc omograp relatonsp s not establsed and te error curve sn t gven out. Background dfferencng metod s used to detect te mage movng target. Gaussan Mxture Model [14, 15] s used to establs a background model to extract background mage and make dfference wt te orgnal mage to obtan a dfference magng, so as to detect te mage movng target. KL algortm s used to track te mage movng target [3-7]. KL s a trackng algortm based on feature ponts, wc conducts multple teratons troug buldng a Gaussan pyramd and predcts te wtdrawn feature ponts, so as to aceve te purpose of trackng. Assumng tat measurement of mage coordnate comples wt Gaussan dstrbuton for te convenence of error analyss, generally, suc assumpton s not verfed [9,10]. Meanwle, measurement standard dfference of omograpc pont on te mage s also from experence and assumpton. us more meanngful pont relatve to specfc value of coordnate converson error s te cange trend of te error dstrbuton curve. e detecton of movng objects n mage utlzes background dfferencng. e background pcture s extracted troug establsng a background model usng Gaussan Mxture Model [14, 15], and s compared wt te orgnal mage to obtan a dfference mage, tus detectng te movng objects n mage. e trackng of movng objects n mage apples KL algortm [3-7]. KL s a tracng algortm based on feature ponts. Predcton of te feature ponts extracted s made troug constructng a Gaussan pyramd and multple teratons, acevng te purpose of trackng. For te measurement of mage coordnates, t s assumed to comply wt te Gaussan dstrbuton, so as to facltate error analyss. s assumpton as generally not been valdated [9, 10]. Meanwle, te standard devaton of measurement of omograpc ponts Fgure 10. Selecton of Homograpc Ponts from te Arport Vdeo Image on mage s also from experence and assumpton. erefore, compared to te specfc value of coordnate converson error, wat s more sgnfcant s te varaton trend of te error dstrbuton curve. Fgure 10 sows te automatc labelng effects aceved by vdeo data and ADS-B data fuson. e track formed by te red crcles n te pcture ndcates te postons of arcraft s ADS-B montorng data mapped nto te mage; te blue rectangle frame ndcates te mage target detected and tracked. e blue pont n te blue rectangle frame ndcates te center Fgure 11. Automatc Labelng n Arport vdeo montorng of mage target. e flgt number of arcraft s marked on te rectangle frame. Fgure 11 sows tat automatc labelng as been aceved for bot flgts CA446 and HU7856 n te vdeo. Due to te lmted vdeo recordng tme and lmted vsual angle of camera n te arport, and tat not all arcrafts are nstalled wt ADS-B arborne equpment, only two flgts CA446 and HU7856 are smultaneously recorded by vdeo and ADS-B n te experment. In order to furter valdate te relablty of coordnate transformaton algortm, a tme frame not recorded p.6

7 ICEA 015 by vdeo s selected and te geograpc coordnates of CA4116 s track as recorded by ADS-B data n te tme frame are converted nto te mage coordnates, ACKNOWLEDGEMENS s paper s supported by te Natonal Natural Scence Foundaton of Cna (Grant No. U13330, U133103). REFERENCES Fgure 1. CA4116 Flgt as sown n Fgure 1. It can be seen from Fgure 1 tat te real measured coordnates of target as provded by ADS-B and converted nto mage coordnates are bascally dentcal to te movng route of arcraft n mage. 6 CONCLUSION (1) e four and above ponts are selected from te vdeo mage and te arport map to correspondngly calculate te omograpy matrx. e omograpy matrx s used to aceve converson between mage coordnates and geograpc coordnates. e ADS-B data and vdeo montorng data ntegraton s used to realze automatc labelng of arcraft n te montorng vdeo. () e coordnate converson error due to exstng coordnate measurement error at te tme of selectng ponts from te mage s calculated, and te relatonsp between pont-correspondng geometrc dstrbuton and error curve s analyzed. e error n te central area surrounded by pont correspondences s relatvely small and te error n te area far away from te pont correspondences s relatvely large. eoretcal dervaton s also dentcal to Monte-Carlo smulaton experment. (3) e ADS-B data and vdeo montorng data actually measured at Cengdu Suanglu Internatonal Arport were used for experments. e results sow tat te metod appled n te paper can automatcally label arcrafts n te arport vdeo montorng n an effectve way. [1] Wang Yong, an Yua. & an Jnwen Vdeo segmentaton algortm wt Gaussan mxture model and sadow removal. Opto-Electronc Engneerng, 35(3): 1-5. [] Zang Quyu, Wang Heng. & Zang Moy, et al. 01. A real-tme and gesture trackng metod based on mxture Gaussan model and mean sft algortm. Journal of Informaton and Computatonal Scence, 9(4): [3] Darby J, Hodson-ole E F. & Costen N, et al. 01. Automated regonal analyss of B-mode ultrasound mages of skeletal muscle movement. Journal of Appled Pysology, 11(1): [4] Omer J F. & Reddng N J GPU-accelerated KL trackng wt monte-carlo-based feature reselecton. // Proceedngs of Dgtal Image Computng ecnques and Applcatons. Canberra;[s.n.]: [5] Manal P, Qong Yang. & Lafrut G, et al Robust low complexty feature trackng. // Proceedngs of te 17t IEEE Internatonal Conference on Image Processng. Hong Kong; IEEE: [6] Km J S, Hwangbo M. & Kanade Realtme affne-potometrc KL feature tracker on GPU n CUDA framework. // Proceedngs of te 1t IEEE Internatonal Conference on Computer Vson Worksops. Kyoto; IEEE Computer Socety: [7] Njad A N, Sara. & E Edrsnge, et al. 01. An Automated Real-me People rackng System Based on KL Features Detecton. e Internatonal Arab Journal of Informaton ecnology, 9(1): [8] He You, Wang Guoong. & Lu Dajn, et al Informaton fuson and applcaton of te mult-sensor. e nd Verson. Bejng: Electronc Industry Press, 007 [9] Hartley R. & Zsserman A, et al. 00. Multple Vew Geometry n Computer Vson. Hefe: Anu Unversty Press. [10]Fad Dornaka. & Fad Cakk. 01. Effcent object detecton and trackng n vdeo sequences. Journal of te Optcal Socety of Amerca. A, Optcs, Image Scence, and Vson, 9(6): [11]Elan Dubrofsky Homograpy Estmaton. Vancouver: e Unversty of Brts Columba. [1]Dmtropoulos K, Grammalds N. &Smtopoulos D, et al Arcraft detecton and trackng usng ntellgent cameras. // Proceedngs of te IEEE Internatonal Conference on Image Processng Genova; IEEE Computer Socety: [13]Wang Guangu, Hu Zany. & Wu Fucao, et al Sngle vew metrology from scene constrants. Image and Vson Computng, 3(9): p.7

8 MAEC Web of Conferences [14]Al Sefdpour. & Nzar Bougula. 01. Spatal color mage segmentaton based on fnte non-gaussan mxture models. Expert Systems wt Applcaton, 39(10): [15]Greggo N, Bernardno A. & Lasc C, et al. 01. Fast estmaton of Gaussan mxture models for mage segmentaton. Macne Vson and Applcatons, 3(4): p.8