Acoustic Camera Image Mosaicing and Super-resolution

Transcription

1 Acoustc Camera Image Mosacng and Super-resoluton K. Km Insttute for Bran and Neural Systems Brown Unv. Box 1843 Provdence RI 091, USA N. Nerett Insttute for Bran and Neural Systems Brown Unv. Box 1843 Provdence RI 091, USA N. Intrator Insttute for Bran and Neural Systems Brown Unv. Box 1843 Provdence RI 091, USA Abstract - Ths paper presents an algorthm for mage regstraton and mosacng on vdeo sequences acqured by an underwater acoustc camera. The sonar mages of our nterest can be characterzed by a hgh nose level, nhomogeneous llumnaton and low frame rate. Imagng geometry of acoustc cameras s sgnfcantly dfferent from that of pnhole cameras. For a planar surface vewed through a pnhole camera undergong translatonal and rotatonal moton, regstraton can be obtaned va a projectve transformaton. We show that, under the same condton, an affne transformaton s a good approxmaton for an acoustc camera. We ntroduce mage fuson algorthms whch reduce the geometrcal dstortons whch are caused by sharp camera movement. Ths reduces mage blur and ncreases mage resoluton. I. INTRODUCTION The acquston of underwater mages s performed n nosy envronments wth low vsblty. For optcal mages, often natural lght s not avalable, and even f artfcal lght s appled, the vsble range s very lmted. For ths reason, sonar systems are wdely used to obtan mages of seabed or other underwater objects. An acoustc camera s a novel devce that can produce a real tme mage sequence. Detaled magng methods of acoustc cameras can be found n [1]. Despte the merts of acoustc cameras over other sonar systems, t stll has shortcomngs compared to normal optcal cameras: ( Lmtaton of sght range: Unlke optcal cameras whch have a -D array of photosensors, acoustc cameras have a 1-D transducer array. -D representaton s obtaned from the temporal sequence of the transducer array. For ths reason, t can collect nformaton from a lmted range. ( Low sgnal-to-nose rato (SNR: The transducer sze s comparable to the wavelength of ultrasonc waves, so the ntensty of a pxel depends not only on the ampltude, but also on the phase dfference of the reflected sgnal. For ths reason, the presence of background nose n underwater envronments results n a Rcan dstrbuton of nose observed n ultrasound mages [13]. The SNR s also sgnfcantly lower than n optcal mages because of the transducer sze. ( Low resoluton wth respect to optcal mages: Due to the large scale of the wavelength of ultrasound compared to wavelength of lght, the number of pxels n the horzontal axs s lmted. (v Inhomogeneous nsonfcaton: Snce one dmenson of the mage s acqured merely based on the relatve sgnal arrval tmes, a sngle nsonfer was used. Consequently, due to the ansotropy of ultrasound radaton from the nsonfer and the specular (mrror-lke surface property, the nsonfcaton s nhomogeneous n acoustc camera mages. The above lmtatons can be addressed by mage mosacng, whch s broadly used to buld a wder vew mage [, 3, 4, 5], or to estmate the moton of a vehcle. For ordnary mages, mosacng s also used for mage enhancement such as denosng, deblurrng, or super- resoluton [6]. In ths paper, we descrbe a mosacng algorthm for a sequence of acoustc camera mages. We show that an affne transformaton s approprate for mages taken from an acoustc camera undergong translatonal and rotatonal moton. We propose a method to regster acoustc camera mages from a sequence usng a feature matchng algorthm. Based on the parameters of mage regstraton, a mosac mage s bult. Durng ths process, the mage qualty s enhanced n terms of SNR and resoluton. II. IMAGING GEOMETRY The transformaton between two acoustc camera mages can be calculated by puttng one mage nto the coordnate system where the mage s on the xy-plane wth the postve y-axs along the center lne of the mage and the center of the arc at the orgn (Fg. 1. Durng the magng process, a pont denoted by a poston vector x = (x, y, z T s projected to the polar coordnates (r, a as follows: u = r = x + snα 1 tan β, v = r α = y + β cos 1 tan, (1.1 where r xy (x + y 1/, or to the Cartesan coordnates (u, v, r = x, α = 1 sn x / rxy, (1. where β s the angle between x and the magng plane. 1

2 III. METHODOLOGY Fg. 1. The magng geometry of an acoustc camera. The camera s located at the orgn of the xyz-coordnate system wth the ptch, yaw, and roll (0, 0, 0. In the next frame (x'y'z'-coordnate, the camera s dsplaced by δx = (δx, δy, δz T and rotated by (φ, θ, ψ. When the camera s translated by δx = (δx, δy, δz T and rotated by (φ, θ, ψ, the new coordnates of x are ( x y z T φθψ ( δ x' = ', ', ' = R x x (1.3 where R φθψ s a 3-by-3 rotaton matrx. The lnear transformaton T between two mages should satsfy The typcal four steps of mage regstraton are: feature detecton, feature matchng, transformaton estmaton, and mage resamplng and transformaton [7]. Feature detecton s the process of fndng objects such as corners, edges, lne ntersectons, etc., manually or automatcally. The features from the sensed mage are pared wth the correspondng features n the reference mage n the second step. In the thrd step, the transformaton s estmated based on the dsplacement vector of each feature. Once the mappng s establshed, the multple mages are combned to generate a mosac mage. In our work, we have found that hgh curvature ponts can be useful as features of nterest n acoustc camera mages. The sum of squared dfference s used to measure the dssmlarty between two mages n the second step. Transform parameters are estmated va a random samplng based method. After the parameters of the affne transform are obtaned, all mages are combned by two methods, so that t maxmzes ether the sgnal-to-nose rato or the lkelhood of the fuson mage. u' x' 1+ tan β ' x 1+ tan β u v' y' 1 tan β ' y 1 tan β = + = T + = T v (1.4 where β' = tan -1 z'/(x' +y' 1/. When the reflectng ponts of the target object are located roughly on a plane such as the sea floor, z can be approxmated by z = ax+ by+ z 0 (1.5 For a, b, β and β' that are suffcently small so that ther squares are neglgble, we have T = R11 + R13 a R1 + R13b R11 x+ R1 y+ R13 z z0 R1 + R3a R + R3b R1 x+ R y+ R3 z z ( δ δ ( δ ( δ δ ( δ (1.6 Ths serves as a frst order approxmaton of the transform between two acoustc camera mages. The error due to the second and hgher order terms may appear as blurrness n the resultng mages. The sx unknown parameters of the affne transform can be obtaned by matchng features n two mages. However, other parameters such as R j, a, b, or δx n (1.6 cannot be fgured out separately because those parameters are coupled and under-constraned. Consequently, under the above approxmaton, t s mpossble to reconstruct the precse egomoton of the acoustc camera merely based on mage regstraton parameters. A. Coordnate mappng and nhomogeneous nsonfcaton equalzaton In order to restore the spatal homogenety of the mage, a transformaton to the Cartesan coordnates has to be performed. Due to the fact that the feld of vew n the angular coordnate of dfferent sensors does not overlap, the resultng pxel sze n the Cartesan coordnates s not homogeneous. Therefore, nearest neghbor nterpolaton was appled to fll the gaps n the mage n the Cartesan coordnate system. Due to the acoustc acquston of mages whch was performed by nsonfyng the area wth a sngle source, an nhomogeneous ntensty profle s obtaned. Ths has to be corrected for effcent mage regstraton and mosacng. For example, Rzhanov et al. have subtracted a D polynomal splne of the mage from the orgnal mage [3]. Prevous work on separaton of llumnaton from reflectance was based on the Retnex theory [8]; the Retnex theory was desgned for optcal mages wth low nose. Usng a homomorphc flterng method wth a Gaussan retnex surround [9], Jobson et al. estmated the llumnaton of an mage, and reconstructed the mage under unform llumnaton. Whle nose s stronger wth an acoustc camera, we demonstrate below that, when ncludng the nose term n the model, the sum of squared dfference s stll a good dssmlarty measure after the retnex rendton. The nosy mage s modeled by I = L Iˆ ( u + η (1.7 G where I(u s the observed mage, L(u the nsonfcaton ntensty, Î(u the normalzed mage under unform nsonfcaton, and η G (u a Gaussan nose wth standard devaton σ G at u. The estmated nsonfcaton ntensty L' s calculated by applyng a Gaussan flter to the orgnal mage, L' (u = I(u * exp (- u /σ L and

3 the estmated unform nsonfcaton mage s ( L I ( ˆ η ( ˆ η u = I u + I + L' u L' u L' u ( ( (1.8 The sum of squared dfference between two unform nsonfcaton mages s = + ( ˆ ( ˆ η η u L' L' SSD I I d u 1, 1 d u 1 (1.9 The second ntegral n (1.9 s ndependent of the true mage, and may be regarded as a constant, provded the nose s unform. A regularzaton factor that s added to L' prevents erroneously excessve ntensty from the speckles n low nsonfcaton regons. B. Feature detecton and putatve matchng Feature detecton and matchng are computatonally demandng. A Gaussan pyramd algorthm has been proposed as a multscale approach for effcent feature detecton and matchng [10, 7]. Magnfed pxels result n jagged edges n the mapped mage. In our mages, feature detecton at the thrd level of the Gaussan pyramd reduces false detecton of corners at the jagged edges. Feature detecton and putatve matchng s ntalzed by translatonal dsplacement detecton. Translatonal dsplacement between the sensed mage and the reference mage s calculated by an exhaustve search on the fourth level of the Gaussan pyramd. Ths process drastcally reduces the area of exhaustve search. After translaton s estmated, hgh curvature ponts of the sensed mage are detected usng the Harrs corner detector [11]. Those ponts are matched to the correspondng ponts n the reference mage by another exhaustve search on the thrd level of the Gaussan pyramd. C. Transform estmaton Image changes due to the sonar system movement are modeled by an affne transformaton as derved n the prevous secton. The affne transformaton descrbes the mage changes by yaw, small ptch and roll and translatonal movement of the sonar system. Ths s vald when multple objects are not present at the same range and angle, whch s the case wth the great majorty of mages n our dataset [1]. The detaled procedure of the algorthm s as follows: ( Feature ponts estmaton: Usng the Harrs corner detector, compute 50 nterest ponts n a preprocessed acoustc camera mage. ( Correspondence search: For a square patch around each feature pont n the sensed mage, fnd the dsplacement n the next mage, usng a cross-correlaton based matchng. ( Transform parameter estmaton: Repeat the followng (1-(3 for 1000 samples. (1 Select 3 putatve matchng pars. ( Usng the matchng pars, estmate the parameters of the affne transform. (3 Fnd the nlers of the estmated transform, and repeat ( wth the nlers untl the estmated nlers are stablzed. (v Set a certan k percentle to defne a threshold n of feature ponts. Then, fnd the n pars of ponts that are closest to each other. The least mean squared error of the pars s used as the crteron. We use the crteron of least square error of k % of samples, where k s emprcally determned. It s smlar to the least-medan of squares (LMS, but t dffers n that t can have a lower breakdown pont (k nstead of 0.5 of LMS, and t uses the mean squared error nstead of the k percentle as the measure of error. It works well wth only a few feature pont pars wth hgh percentage of outlers. In addton, t yelds a measure of goodness of the transformaton, whch helps to decde whether to contnue mosacng or to stop, for example when the rsk of msmatch s hgh. D. Selectve mage mosacng Estmatng the geometrc transformaton between consecutve frames may not be the optmal mage regstraton. Ths s due to the fact that the camera s movng and consequently, dstorton between two consecutve frames vares by sgnfcant amount. Therefore, we compare the relatve dstorton between consecutve frames, and start the mage regstraton from a frame that s relatvely mnmally dstorted wth respect to all other frames. We then utlze n the mage regstraton and enhancement process only the subset of frames whch are mnmally dstorted wth respect to the reference frame. Ths leads to smaller geometrc dstortons, and therefore mage enhancement n addton to the mosac effect. E. Mosacng and resoluton enhancement va mage fuson After the regstraton, a mosac mage s constructed. Snce the nose s present regardless of the nsonfcaton condton, t can deterorate the mosac mage f not treated properly. For example, f we average well-nsonfed mages and poorly-nsonfed mages, the SNR wll be deterorated because nose may accumulate. In ths case, mosacng va averagng can be descrbed as the followng relatonshp: I mosac 1 N N = 1 ( I ( u = u (1.10 In ths case, the SNR of the mosac mage s SNR L mosac ( u = (1.11 where L (u s the nsonfcaton ntensty of the -th σ 3

4 mage at u. Note that the SNR s a functon of u because the nsonfcaton ntensty vares wthn the mage. Regons wth poor nsonfcaton receve lower weght n the averagng. Denote the weght of the -th mage by α (u, where Σ α (u = 1. Then, the mosac mage s, I ( = α ( I ( u u u (1.1 mosac of whch the SNR mosac s, SNR ( u; α,, α mosac N ( α ( L ( σ α ( u u L u = σ (1.13 Equalty holds when α (u = L k (u/σl k (u, where SNR mosac s maxmzed. F. Maxmum-lkelhood estmaton of equalzed mage Together wth the weghted mosac mage, the mage qualty can be enhanced by an addtonal process. Accordng to the nhomogeneous nsonfcaton model, the ntensty of a reflected sgnal can vary dependng on the nsonfcaton condton. Gven the equalzed pxel values Ĩ 1 (u,, Ĩ N(u under the nsonfcaton ntensty L 1 (u,, L N (u, the equalzed true pxel value of reflecton ntensty at u s estmated by ( I I ( ( ˆ( ( ( 1 N ( ( 1,..., N ( P I, I,..., I Iˆ u = arg max P I u = I u I u, I u,..., I u mle P I I I P I = arg max ( 1 N (1.14 and, by approxmatng the Rcan nose dstrbuton by a Gaussan, we have ( I 1 I σ / L Iˆ mle = arg max exp... I 1 ( I N I exp P ( I σ / LN (1.15 where the denomnator of (1.14 s neglected. By combnng the exponents, fnally, ( I 0 I σ Iˆ mle = arg max log P( I, I 0 where 1 1 = σ σ 0 L, ( ( ( ( L u I u L u I u 0 = = L L I. (1.16 (1.17 Note that the maxmum-lkelhood estmaton result has the same numerator as n (1.1, but the denomnator s the sum of square of the nsonfcaton ntensty. Ths result s useful snce we can choose an approprate pror P(I(u to get the best a posteror estmaton of the equalzed mage. IV. RESULTS The algorthm was tested on a boat wreckage sequence. A DIDSON system scanned a shpwreck at approxmately 30 meters depth for 85 seconds and took 446 frames of mages. About 40 frames among them show the vessel from head to stern, and another 40 frames show t from stern to head. The algorthm was appled to those two sub-sequences to buld two mosac mages. Fg.. Features from a sensed mage are pared wth correspondng ponts n the reference mage. The outlers (features wth weaker matchng are defned by those pars wth hgher matchng error after the estmated transformaton. Fg. depcts two consecutve acoustc mages together wth a set of matched (red and non-matched (black feature ponts. These matched feature ponts n the reference mage, whch were found usng the cross-correlaton of patches around the feature ponts n the sensed mage, are used to estmate the geometrc transformaton between the two mages. Cross-correlaton was found to be more robust than a conventonal approach [1] n whch features are ndependently found and matched between the two mages. Ths s a consequence of the hgh nose n the mage and the fact that the exact locaton of the features s not so well defned. Fg. 3 represents the man result of the paper a mosac mage of multple acoustc mages. The mosaced mage contans nformaton whch spans multple frames, each frame correspondng to a small porton of the nsonfed object. The combnaton mages whch have been transformed to be n the same coordnate system provde subpxel mage resoluton enhancement. Fg. 4(b shows detal of the target before mosacng. The resoluton enhancement follows from the fact that one 4

5 pxel n the orgnal polar coordnate system s mapped to multple pxels wth the same ntensty n the Cartesan coordnate system. Dfferent frames lead to partal overlap of these multple pxels, so that after averagng, a subpxel resoluton s acheved (See Fg. 4(a. (a (b Fg. 4. Resoluton enhancement by averagng mages. (a A mosac mage of 9 consecutve frames followed by a geometrc transformaton. (b A sngle frame from the same vew as (a. (a (b Fg. 3. Demonstraton of weghted averagng effect: Panel (a demonstrates the unform average of the whole mages, whle panel (b shows the weghted average, n whch nsonfcaton profle was utlzed durng averagng. Averagng of dfferent acoustc mages after brngng them to the same coordnate system (same vewpont leads to the classcal effect of denosng. Ths s clearly seen n Fg. 3 on the whole target, and n partcular n the comparson of a small porton of the target n Fg. 4. Fg. 3(a and (b depct the same target from a dfferent mage sequence, but (b utlzng the nsonfcaton profle durng averagng. Fg. 5 shows the maxmum-lkelhood estmaton of the equalzed mage. Ths mage has the closest pxel values to the truth mage regardless the nsonfcaton condton. IV. CONCLUSION Acoustc camera technology s becomng essental for underwater exploraton n nosy envronments wth low vsblty. The acoustc camera, wth ts specfc sensor desgn, poses some challenges n terms of mage resoluton, nose removal and area coverage. Fg. 5. Maxmum-lkelhood estmaton of the equalzed mosac mage. Red area ndcates saturated pxels. In ths paper, we have presented a complete algorthm to acheve mage mosacng, denosng and resoluton enhancement from a sequence of acoustc camera mages. We descrbed the steps that were requred to acheve ths mosacng. Ths ncluded 5

6 modelng the specfc geometry of acoustc camera mages whch sharply dffers from pnhole camera geometry. The dfferent geometry, and n partcular, the fact that the mages are acqured n a polar coordnate system, complcates the search and matchng of feature ponts n consecutve mages. Moreover, n ths partcular geometry, pxels n the polar coordnate system are mapped to a collecton of pxels wth the same ntensty n the Cartesan coordnate system. Snce consecutve mages were taken from dfferent vewponts, a subpxel enhancement effect was acheved n the process of averagng n addton to the denosng effect. We have presented a novel method n whch features extracted by a certan algorthm are locally matched to the reference mage va cross-correlaton. Ths method was found to be more robust than a conventonal approach n whch features are ndependently found and matched between two mages. In partcular, ths s more pronounced when the number of pxels avalable for feature comparson s lmted. The mage fuson method requres the regstraton between mages to be globally consstent; otherwse, artfacts due to cumulatve regstraton errors may appear as dscussed n [5]. In cases where the global regstraton s dffcult to acheve, for example, due to the vewng angle change, mages are locally regstered wth only neghborng frames. Ths method s not drectly applcable to the acoustc camera sequences, because the resoluton enhancement obtaned from the global mosacng s not neglgble. In subsequent work, the trade-off between the global consstency of regstraton and the resoluton enhancement wll be further nvestgated. IEEE Journal of Oceanc Engneerng, 8(4:651 67, 003. [6] D. Capel and A. Zsserman. Computer vson appled to super-resoluton, IEEE Sgnal Processng Magazne, 0(3:7 86, 003. [7] B. Ztová and J. Flusser. Image regstraton methods: a survey, Image and Vson Computng, 1: , 003. [8] E. H. Land and J. J. McCann. Lghtness and the retnex theory, Journal of the Optcal Socety of Amerca, 61:1 11, [9] D. J. Jobson, Z. Rahman, and G. A.Woodell. Propertes and performance of the center/surround Retnex, IEEE Transactons on Image Processng, 6:451 46, [10] D. A. Forsyth and J. Ponce. Computer Vson: A Modern Approach. Pearson Educaton, Inc., Upper Saddle Rver, NJ, 003. [11] C. J. Harrs and M. Stephens. A combned corner and edge detector, n Proceedngs on 4 th Alvey Vson Conference, pages , [1] Z. Zhang, R. Derche, O Faugeras, and Q.-T. Luong. A robutst technque for matchng two uncalbrated mages through the recovery of the unknown eppolar geometry, Artfcal Intellgence, 78:87 119, [13] R. F. Wagner, S. W. Smth, J. M. Sandrk, and H. Lopez. Statstcs of speckle n ultrasound B-scans, IEEE Transactons on Soncs and Ultrasoncs, 30(3: , ACKNOWLEDGMENT Ths work was partly supported by ONR grant N C-096. We thank E. O. Belcher for provdng full detals about the data. Leon N Cooper and other members of IBNS have provded valuable comments. REFERENCES [1] E. O. Belcher, B. Matsuyama, and G. M. Trmble. Object dentfcaton wth acoustc lenses, n Proceedngs of Oceans 01 MTS/IEEE, pages 6 11, 001. [] R. L. Marks, S. M. Rock, and M. J. Lee. Real-tme vdeo mosackng of the ocean floor, IEEE Journal of Oceanc Engneerng, 0(3:9 41, [3] Y. Rzhanov, L. M. Lnnet, and R. Forbes. Underwater vdeo mosacng for seabed mappng, n Internatonal Conference on Image processng, volume, pages 4 7, 000. [4] E. Trucco, Y. R. Petllot, I. Tena Ruz, K. Plakas, and D. M. Lane. Feature trackng n vdeo and sonar subsea sequences wth applcatons, Computer Vson and Image Understandng, 79:9 1, 000. [5] O. Pzarro and H. Sngh. Toward large-area mosacng for underwater scentfc applcatons, 6