MULTIPLE OBJECT DETECTION AND TRACKING IN SONAR MOVIES USING AN IMPROVED TEMPORAL DIFFERENCING APPROACH AND TEXTURE ANALYSIS

Transcription

1 U.P.B. S. Bull., Seres A, Vol. 74, Iss. 2, 2012 ISSN MULTIPLE OBJECT DETECTION AND TRACKING IN SONAR MOVIES USING AN IMPROVED TEMPORAL DIFFERENCING APPROACH AND TEXTURE ANALYSIS Tudor BARBU 1 Lurarea de faţă abordează domenul detetăr ş urmărr obetelor vdeo în sevenţele de tp sonar. Într-o prmă fază sunt aplate anumte operaţun de preproesare a sevenţe vdeo analzate, preum o elmnare a zgomotulu bazată pe euaţ u dervate parţale ş o îmbunătăţre a ontrastulu. În ontnuare regunle supuse mşăr sunt detetate automat prn ntermedul une metode îmbunătăţte de dferenţere a adrelor. Câteva operaţ matemate morfologe sunt utlzate în sopul dentfăr prnpalelor obete aflate în mşare. O tehnă de urmărre a obetelor vdeo multple, utlzând o analză a textur bazată pe fltrarea Gabor 2D a obetelor magste ş un proes de potrvre a aestora, este apo propusă. Ths paper approahes the omputer vson tas of vdeo obet deteton and trang n sonar move sequenes. Some pre-proessng operatons, suh as a PDE based vdeo denosng and a ontrast enhanement, are performed on the analyzed sequene, frst. Then, the movng regons are deteted automatally usng an mproved frame-dfferene based approah. Some mathematal morphologal operatons are used to dentfy the man movng obets. A multple vdeo obet trang tehnque, usng a 2D Gabor flter-based texture analyss of mage obets and an obet mathng proess s then provded. Keywords: sonar vdeo sequene, PDE mage denosng, movng obet, obet deteton, temporal dfferenng, morphologal operatons, 2D Gabor flterng, texture analyss, template mathng. 1. Introduton Vdeo obet deteton and trang represents an mportant and hallengng omputer vson doman. Obvously, t onssts of two losely related vdeo analyss proesses. Thus, vdeo obet deteton nvolves loatng an mage obet n the frames of a vdeo sequene, whle vdeo trang represents the proess of montorng the vdeo obet spatal and temporal hanges durng the move sequene, nludng ts presene, poston, sze, shape, et. A vdeo obet trang 1 Senor Researher II, Insttute of Computer Sene, Romanan Aademy, Ias branh, Romana, e-mal: tudbar@t.tuas.ro

2 28 Tudor Barbu approah has to solve the temporal orrespondene problem that s the tas of mathng the target obet n suessve vdeo frames [1]. Usually, the trang proess starts wth detetng the ntal nstane of the vdeo obet, then dentfyng that mage obet repeatedly n subsequent frame sequene. Vdeo trang s often a dffult proess, due to some fators suh as abrupt obet moton, obet olusons and amera moton. Numerous vdeo deteton and trang tehnques have been developed n reent years. Obet deteton an be performed usng many approahes, suh as: regon-based mage segmentaton [2,3], baground subtraton [4], temporal dfferenng [5], atve ontour models [6], generalzed Hough transforms [7] and template mathng [8]. Also, vdeo obet trang has been approahed through varous tehnques, suh as: statstal methods, le those based on Kalman flterng or Hdden Marov Models [9], orrelaton-based obet mathng, ernel trang [10], optal flow [11] or ontour trang. Obet dentfaton and trang has a wde varety of omputer vson applaton areas. The most mportant of them are the vdeo ompresson, vdeo survellane, human-omputer nteraton, vdeo ndexng and retreval, medal magng, augmented realty and robots. We propose an automat vdeo obet deteton and trang system for multple movng obets n sonar vdeo sequenes n ths artle [12]. The automat harater of the system means that vdeo obets are deteted and traed automatally, no nteratvty beng present. For example, the user wll not be ased to selet an obet from a frame to be traed by the system. Also, the proposed system wors for movng obets only and not any vdeo obets. Thus, our vdeo analyss tehnques are appled on fxed amera move sequenes. Also, the proposed approahes provde the best deteton and trang results when appled on ultrasound vdeos [12]. The sonar move sequenes onsdered for deteton and trang represent vdeo senes ontanng several obets (more than one) movng on a qute empty baground. The ultrasound vdeo sequene must pre-proessed before vdeo analyss, usng some nose removal and frame enhanement operatons whh are desrbed n the next seton [13,14]. Then, a novel movng obet deteton approah s proposed n the thrd seton. It uses an mproved temporal dfferenng tehnque that represents the man ontrbuton of ths paper. The vdeo trang approah provded n the fourth seton s based on a novel ontent-based obet mathng tehnque. The Gabor flterng based texture analyss approah used by the obet mathng proess s another orgnal and mportant ontrbuton of ths artle [15]. The performed experments are mentoned n the ffth seton and the paper ends wth a seton of onlusons.

3 Multple obet deteton and trang n sonar moves [...] approah and texture analyss Sonar move preproessng We proposed some sonar mage and vdeo denosng tehnques n our prevous wors [13,14]. Thus, we developed a PDE based spele nose removal model that s expressed as follows: I = Δ [ f ( Δ I ) Δ I ] t, [ 1, n] (1) x f ( x ) = 2 x + where { I 1,..., I n } s the nosed vdeo sequene, Δ I = 2 I and f represents a monotone dereasng nose smoothng funton [13]. Other nose removal operatons, suh as medan flterng and Wener adaptve flterng, an be further appled on the PDE fltered mage. The denosed vdeo s then normalzed to enhane ts ontrast. Sharpenng the ontrast ould help the mage obet deteton proess. We use the followng normalzaton formula for the vdeo sequene: ' It (, ) mn( It ) It (, ) = 255, [1, M ], [1, N ], (2) max( I ) mn( I ) where eah frame I t represents a [ M N ] graysale mage. t t 3. Temporal dfferenng approah for movng obet deteton Let the enhaned sonar move sequene be noted as Vd = { I 1,..., I n }, where I ould represent here the vdeo eyframes, to smplfy the omputaton proess. We propose an mproved temporal dfferenng (frame-dfferene based) algorthm for movng obet deteton. We assume that the movng vdeo obets from Vd do not ollde, are not very lose to eah other and do not stop untl the they reah the last frame. The graysale mage obtaned as dfferene of two suessve frames ndates the vdeo moton between them, ts non-bla regons representng the movng regons. Unfortunately, a movng regon does not represent always an entre movng obet. An mage obet n a sonar frame an be omposed of more homogeneous regons, haraterzed by varous ntenstes. Some of them have hgher ntenstes than that of the mage baground, whle other regons have lower ntenstes. A fltered ultrasound mage (frame) s haraterzed by a large and homogeneous baground. We onsder the followng notaton for the dfferene between two vdeo frames:

4 30 Tudor Barbu DF (, ) = I I, [1, n ], (3) Any movng vdeo obet that s present n the frames I and I, s represented n mage DF(, ) by some non-bla regons orrespondng to ts homogeneous regons. The hgh-ntensty regons are dsplayed n DF(, ) at the loatons ouped n I, whle low-ntensty regons are dsplayed n DF (, ) at ther postons n I. Suh a vdeo frame dfferene example s dsplayed n Fg. 1. Two frames of an enhaned (denosed and normalzed) sonar move are represented n the fgure, together wth ther dfferene mage. One an see n that mage the two lghter zones orrespondng to a hgh ntensty regon and a low-ntensty regon of an obet respetvely. Fg. 1. Temporal dfferenng for two sonar frames Frst, we propose an algorthm that determnes the movng mage obets from I 1 to I n 1. Then, a speal reasonng s used for the last frame. At eah step, our algorthm detets the movng obets of the frame I, usng the next vdeo frames, I + 1 and I + 2. It omputes the vdeo frame

5 Multple obet deteton and trang n sonar moves [...] approah and texture analyss 31 dfferenes DF (, + 1) and DF (, + 2) frst. Then, these frame dfferenes are onverted nto the bnary form, by settng to 1 all ther non-zero pxels (or all pxels exeedng a gven threshold). Let the resulted bnary mages be noted as DF B (, + 1) and DF B (, + 2) respetvely. Next, some mathematal morphologal operatons are appled to them [16]. We onsder a morphologal losng proess, representng a dlaton followed by an eroson, to be performed on both the bnary mages. We obtan the followng losed mages: DFB (, + 1) = DF DFB (, + 2) = DF B B (, + 1) D (, + 2) D (4) where represents the losng operator and D s the hosen struturng element, representng a ds wth radus 1. The onneted omponents of the losed bnary mages are determned, the omponents havng very small areas (under a seleted threshold) beng dsarded. The two morphologally proessed frame dfferene mages DF B (, + 1) and DF B (, + 2), orrespondng to a vdeo frame of a sonar move sequene ontanng mltary vehles, are desrbed, as an example, n Fg. 2. One omputes the nterseton of the bnary mages obtaned from (4). Heren we onsder the nterseton of two bnary mages to be the mage havng the pxel values of the two bnary mages where they onde and 0 n the loatons where the two mages dffer. Therefore, the nterseton mage s obtaned as: Int ( ) = DFB (, + 1) DFB (, 2) (5) 1 + whh means Int 1 ( DFB (, + 1)[ x, y] = DFB (, + 2)[ x, y] )[ x, y] = 0, DFB (, + 1)[ x, y] DFB (, + 2)[ x, y] (6)

6 32 Tudor Barbu Fg. 2. Morphologally proessed frame dfferene mages example The onneted omponents of Int 1 ( ) orrespond to all hgh-ntensty movng regons of I. Therefore, the hgh-ntensty movng zones of the sonar frame I from Fg. 2 are those orrespondng to the whte regons from the next fgure, obtaned from (6).

7 Multple obet deteton and trang n sonar moves [...] approah and texture analyss 33 Fg. 3. Loatons of the hgh-ntensty regons of movng obets The low-ntensty movng regons of the vdeo frame are determned usng a smlar dentfaton algorthm. Thus, these regons orrespond to the onneted omponents of the bnary nterseton mage Int ( ) = DF ( + 1, ) DF ( 2, ) (7) 2 B B + where the losed bnary frame dfferenes DF B ( + 1, ) and DF B ( + 2, ) are omputed as n the prevous ase. Thus, the low-ntensty regons of the frame I from Fg. 2 are those orrespondng to the onneted omponents of Fg. 4, obtaned from relaton (7). Fg. 4. Loatons of the low-ntensty regons of movng obets

8 34 Tudor Barbu The bnary mage representng the sum of these two mage ntersetons s omputed as: Mov ) = Int ( ) + Int ( ) (8) ( 1 2 The onneted omponents of Mov() orrespond to the movng obets of the frame I. If the summng relaton (8) s appled to bnary mages n Fg. 3 and Fg. 4, the mage dsplayed n Fg. 5 s obtaned. The two onneted omponents represent the loatons of the movng obets of the frst sonar vdeo frame from Fg. 2. Fg. 5. Loatons of the movng obets of For I 1,..., I n 1 the movng vdeo obets are deteted n the same way. The last step of the deteton tehnque onssts of dentfaton of the fnal loatons n I n of the movng obets. Let us name these zones of I n as movng obets of the frame, although ths term somewhat norret here. Ther deteton s performed through a baward proess gven by the relaton Mov n) = Int ( n) + Int ( ), where ( 1 2 n I and Int1( n) = DFB ( n, n 1) DFB ( n, n 2) (9)

9 Multple obet deteton and trang n sonar moves [...] approah and texture analyss 35 Int 2 ( n) = DFB ( n 1, n) DFB ( n 2, n) (10) As the result of the proposed proedure, there are K deteted mage obets n eah vdeo frame. The next step s the trang of these obets n the vdeo sequene. 4. Vdeo obet trang usng a texture analyss approah In ths seton we propose a vdeo trang tehnque for the deteted movng obets. So, we onsder an obet mathng approah usng the texture analyss of the K mage obets dentfed n eah frame. One nows the movng mage obets of eah frame, but the nstanes of any gven obet n the next frames reman unnown. Our vdeo trang approah fnds for eah mage obet of a frame ts nstane n the next frame, representng the orrespondent movng mage obet. In our approah the mathng obet has to be the most smlar obet from the next frame. Therefore, the onsdered trang method s based on an mage obet reognton proess. The frst step of ths reognton proess s a ontent-based feature extraton for mage obets. The seond step onssts of a supervsed feature vetor lassfaton based on a mnmum dstane lassfer. We note the sequene of the movng mage obets from the vdeo frame I as: { Ob,..., Ob }, [1, ] Ob ( ) = 1 n (11) these obets beng numbered n the mage from left to rght and from up to down. Any traed vdeo obet s modeled as an ordered sequene of smlar mage 1 n obets, [ Ob,..., Obpos (, ),..., Obpos (, n) ], where the ontent smlarty s expressed as: 1 pos(, ) K Ob... Ob... Ob ) (12) n pos(, n where pos(, ) [1, K]. We do not onsder a shape smlarty between these obets, beause the sonar mage obets may often have smlar shapes. We onsder a texture-based ontent smlarty for vdeo analyss nstead. For eah movng obet Ob one determnes the sub-mage Im ( Ob ) orrespondng to ts boundng box. A texture analyss s performed for eah submage. Thus, we propose a texture-based ontent desrptor usng 2D Gabor flters for the analyzed sub mage [15].

10 36 Tudor Barbu Two-dmenson Gabor flterng represents a wdely used tool n mage texture analyss and many other mage proessng and analyss domans [15,17]. The Gabor flter onsttutes a band-pass lnear flter whose mpulse response s defned by a harmon funton multpled by a Gaussan funton. The two-dmensonal Gabor flter an be vewed as a snusodal plane of partular frequeny and orentaton, modulated by a Gaussan envelope. We onsder an even-symmetr 2D Gabor flter, havng the followng form: G θ 2 2 x y θ θ = + os( 2πf xθ ) 2 (13) σ x σ y, f x y, σ x, σ, ) exp y ( 2 where xθ = xsn θ + y osθ, yθ = xosθ ysnθ, f s the entral frequeny π of the snusodal plane wave at the angle θ = wth the x axs, σx and σ y n are the standard devatons of the Gaussan envelope along the two axes [15]. The mage of eah obet Ob s fltered wth Gθ, f, σ x, σ at varous y orentatons, radal frequenes and standard devatons. We selet some proper flter parameters: σ x = 2, σ y = 1, f { 0.75,1.5 } and π 2π θ,, π. 3 3 G, omposed of 6 θ The resulted 2D Gabor flter ban { }, f,2,1 f { 0.75,1.5}, [1,3] hannels, s appled to the obet mage, by onvolvng t wth eah Gabor flter from the set. The resulted Gabor responses are onatenated nto a 3D feature vetor. The feature extraton proess s modelled as followng: where and V V ( Ob θ ( z ), f ( z ), σ x, σ y y )[ x, y, z] = V ( Ob )[ x, ] (14) θ z, z [1,3] f1, z [1,3] θ ( z) =, f ( z) = (15) θz n, z [4,6] f2, z [4,6] θ ( z), f ( z), σ, ( ( ), ( ),, y x σ y θ z f z σ x σ y Im( Ob ))[ x, y] = Im( Ob )( x, y) G ( x, ) (16)

11 Multple obet deteton and trang n sonar moves [...] approah and texture analyss 37 So, for eah movng obet we get a 3D feature vetor V ( Ob ), that s a robust ontent desrptor. The movng obets are ompared by omputng the Euldan dstane between ther feature vetors. Obvously, the analyzed obets may have varous dmensons, so a reszng proess must be performed on ther mages before applyng the Euldan metr. The vdeo obet trang proess has n 1 steps. At eah step, the mathng algorthm detets the nstanes of all obets Ob n the next frame, I +1. The next nstane of any vdeo obet s the most relevant obet of I + 1, that s the mage obet orrespondng to the mnmum dstane between feature +1 vetors. Thus, one dentfes the mathng obet Ob Ob, where nd nd = arg mn d ( V ( Ob [1, K ] ), V ( Ob + 1 )), [1, n 1], [1, K ] (17) where d represents the Euldean dstane. An example based on ths vdeo obet trang proess s represented n Fg. 6. Obet mathng s appled on the same sequene of the three onseutve vdeo frames analyzed for obet deteton. The deteted obets, two for eah frame, are bounded by olored retangles. The dentfed math of eah obet s mared by the same olor (red or blue). The arrows ndate the traetory of the traed vdeo obet. 5. Experments The vdeo obet deteton and trang tehnques proposed n ths paper have been tested on varous sonar move datasets. We have performed numerous vdeo analyss experments on ultrasound mage sequenes and obtaned satsfatory results, whh prove the effetveness of our tehnques. Hgh deteton and trang rates are produed by the provded approahes. The movng obet dentfaton method has a deteton rate of approxmately 90%. There have been obtaned hgh values for performane parameters of Preson, Reall and F 1. That means there are very few mssed hts (undeteted movng obets) and wrongly deteted mage obets.

12 38 Tudor Barbu Fg. 6. Obet mathng proess n a sonar vdeo The vdeo trang algorthm s haraterzed by a hgh obet reognton rate that s over 80%. The performane parameters are Preson = 0.80, Reall = Ths means there s a small number of false postves and false negatves too. 6. Conlusons We have developed an automat vdeo obet deteton and trang system for ultrasound moves, haraterzed by a fxed amera and the presene of

13 Multple obet deteton and trang n sonar moves [...] approah and texture analyss 39 multple movng obets. Ths paper brngs mportant ontrbutons n both the obet deteton and vdeo obet trang felds. Thus, we have provded a novel movng obet deteton tehnque. Our deteton approah uses an mproved frame dfferene algorthm to detet the vdeo moton. Ths temporal dfferenng method uses the dfferenes between the urrent frame and the next two frames, and some morphologal operatons appled to the bnarzed frame dfferenes. In the vdeo trang stage we have desrbed a novel mage obet mathng method. The proposed obet reognton approah uses a 2D Gabor flterng based feature extraton, that produes robust obet ontent desrptors, and a supervsed mnmum dstane based obet lassfaton approah. The frame dfferene based tehnque and the texture analyss method represent the man orgnal ontrbutons of ths artle. The automat harater of our deteton and trang system s also a very mportant thng, no nteratvty beng requred. The deteton and trang results desrbed here an be suessfully appled n numerous mportant domans suh as vdeo ndexng and retreval, robots, sonar navgaton and vdeo survellane. Our future researh n ths vdeo analyss doman wll fous on mprovng the proposed tehnques. Therefore, we ntend to mae them wor for olldng vdeo obets or some amera motons. Anowledgments The researh desrbed here has been supported by the grant PN II, Programme 4 - Partnershps n prortes domans , Proet type: PC Complex proets, Adbosonar adaptve bo-mmet sonar heads for autonomous vehles, Contrat no: 12079/2008. R E F E R E N C E S [1] A. Ylmaz, Obet Trang: A Survey, ACM Computng Surveys, Vol. 38, No. 4, Artle 13, Deember [2] J. Sh, J. Mal, Normalzed uts and mage segmentaton, IEEE Trans. Patt. Analy. Mah., Intell. 22, 8, pp , [3] T. Barbu, A Pattern Reognton Approah to Image Segmentaton, Proeedngs of the Romanan Aademy, Seres A, Volume 4, Number 2, pp , May-August [4] C. Wren, A. Azarbaean, A. Pentland, Pfnder: Real-tme trang of the human body, IEEE Trans. Patt. Analy. Mah. Intell. 19, 7, pp , [5] R. Jan, H. Nagel, On the analyss of aumulatve dfferene ptures from mage sequenes of real world senes, IEEE Trans. Patt. Analy. Mah. Intell. 1, 2, pp , 1979.

14 40 Tudor Barbu [6] M. Kass, A. Wtn, D. Terzopoulos, Snaes: Atve Contour Models, Internatonal Journal of Computer Vson, pp , [7] P. Lappas, J. N. Carter, R. I. Damper, Robust evdene-based obet trang, Journal Pattern Reognton Letters, Volume 23 Issue 1-3, January 2002, Elsever Sene In. New Yor, NY, USA. [8] X. Hu, Y. Tang, Z. Zhang, Vdeo Obet Mathng based on SIFT Algorthm, n Proeedngs of Internatonal Conferene on Neural Networs and Sgnal Proessng, pp , [9] N. Peterfreund, Robust Trang of Poston and Veloty Wth Kalman Snaes, IEEE Transatons on Pattern Analyss and Mahne Intellgene, vol. 21, no. 6, June [10] D. Comanu, Kernel-based Obet Trang, IEEE Transatons on Pattern Analyss and Mahne Intellgene, vol. 25, no. 5, [11] L. Wxson, Detetng Salent Moton by Aumulatng Dretonally-Consstent Flow, IEEE Transatons on pattern analyss and mahne ntellgene, vol. 22, no. 8, August [12] E. Truo, Y. R. Petllot, I. Tena Ruz, K. Plaas, D. M. Lane, Feature Trang n Vdeo and Sonar Subsea Sequenes wth Applatons, Computer Vson and Image Understandng, 79, pp , [13] T. Barbu, V. Barbu, V. Bga, D. Coa, A PDE varatonal approah to mage denosng and restoraton, Nonlnear Analyss: Real World Applatons, Volume 10, Issue 3, pp , June [14] T. Barbu, A PDE based Model for Sonar Image and Vdeo Denosng, Analele Stntfe ale Unverstat Ovdus, Constanta, Sera Matemata, Volumul 19, Fasola 2, [15] T. Barbu, Content-based mage retreval system usng Gabor flterng, Proeedngs of the 20th Internatonal Worshop on Database and Expert Systems Applatons Mnng and Management, pp , 31 August - 4 September, Lnz, Austra, [16] J. Serra, P. Solle, Mathematal Morphology and Its Applatons to Image Proessng, Proeedngs of the 2 nd nternatonal symposum on mathematal morphology (ISMM'94), [17] A. Jan, N. Ratha, S. Lashmanan, Obet deteton usng Gabor flters, Pattern Reognton, vol. 30, pp , 1997.