Non-iterative Construction of Super-Resolution Image from an Acoustic Camera Video Sequence

Transcription

1 CIHSPS IEEE Internatonal Conference on Computatonal Intellgence for Homeland Securty and Personal Safety Orlando, FL, USA, 3 March Aprl 005 Non-teratve Constructon of Super-Resoluton Image from an Acoustc Camera Vdeo Sequence K. Km, N. Nerett, N. Intrator Insttute for Bran and Neural Systems, Brown Unversty, Provdence RI 09, USA Phone: +40) , Fax: +40) , E mal: ko@brown.edu. Abstract The low vsblty condton of underwater envronments s often a lmtng factor n the exploraton of rvers or coastal areas. In such stuatons, relatvely hgh resoluton mages produced by an acoustc camera can be an nformatve measure for the detecton of terran and objects. In ths paper, we present a non-teratve super-resoluton algorthm for the constructon of a hgh resoluton mage from multple frames of acoustc camera mages. We dscuss the geometry of nsonfcaton and mage acquston of an acoustc camera, and descrbe the detaled procedures for the regstraton, nsonfcaton correcton, and fuson. Keywords acoustc camera, mosacng, regstraton, vdeo enhancement, super-resoluton. I. INTRODUCTION An acoustc camera s a hgh resoluton ultrasonc magng devce that can produce underwater vdeo sequences, very useful for the purpose of the survellance of coastal areas []. It can be mounted on an autonomous underwater vehcle AUV) or carred by a dver, and can be sent out to explore potentally harmful objects. Despte the merts of acoustc cameras over other sonar systems, t stll has shortcomngs compared to normal optcal cameras: ) Lmtaton of sght range: Unlke optcal cameras whch have a -D array of photosensors, acoustc cameras have a - D transducer array. -D representaton s obtaned from the temporal sequence of the transducer array. For ths reason, t can collect nformaton from a lmted range. ) Low sgnal-to-nose rato SNR): The transducer sze s comparable to the wavelength of ultrasonc waves, so the ntensty of a pxel depends not only on the ampltude, but also on the phase dfference of the reflected sgnal. For ths reason, the presence of background nose n underwater envronments results n a Rcan dstrbuton of nose observed n ultrasound mages []. The SNR s also sgnfcantly lower than n optcal mages because of the transducer sze. ) Low resoluton wth respect to optcal mages: Due to the large scale of the wavelength of ultrasound compared to the wavelength of lght, the number of pxels n the horzontal axs s lmted. v) Inhomogeneous nsonfcaton: Snce one dmenson of the mage s acqured merely based on the relatve sgnal arrval tmes, a sngle nsonfer was used. Consequently, due to the ansotropy of ultrasound radaton from the nsonfer, the nsonfcaton s nhomogeneous n acoustc camera mages. The above lmtatons can be addressed by mage mosacng, whch s broadly used to buld a wder vew mage [3], [4], [5], or to estmate the moton of a vehcle [6], [7]. For ordnary mages, mosacng s also used for mage enhancement such as denosng, deblurrng, or super-resoluton [8], [9]. In ths paper, we descrbe the mosacng of a sequence of acoustc camera mages, and present a new non-teratve algorthm for the constructon of a hgh resoluton mage from the mage sequence. II. IMAGING GEOMETRY The transformaton between two acoustc camera mages can be calculated by puttng one mage nto the coordnate system where the mage s on the xy-plane wth the postve y-axs along the center lne of the mage and the center of the arc at the orgn Fg. ). Durng the magng process, a pont denoted by a poston vector x = x, y, z) s projected to the polar coordnates r, a) as follows: r = x, α = sn x r xy, ) where r xy x + y ) /, or to the Cartesan coordnates u = u, v),

2 u and v can then be rewrtten as z = ax + by + z 0. 5) u z ) = + x + y { R + R 3 a) x + R + R 3 b) y Fg.. The magng geometry of an acoustc camera. The camera s located at the orgn of the xyz-coordnate system wth the ptch, yaw, and roll 0. In the next frame x y z -coordnate), the camera s dsplaced by δx = δx, δy, δz) and rotated by φ, θ, ψ). u = r sn α = x + tan β ) v = r cos α = y + tan β, 3) where β s the angle between x and the magng plane. When the camera s translated by δx = δx, δy, δz) and rotated by φ, θ, ψ), the new coordnates of x are x = x, y, z ) = R φθψ x δx) 4) where R φθψ s a 3 3 rotaton matrx. The lnear transformaton T between two mages should satsfy u v = = T = T x + tan β y + tan β x + tan β y + tan β u v, where tan β = z /x + y ) /. When the reflectng ponts of the target object are located roughly on a plane such as the sea floor, z can be approxmated by R φθψ = cos φ cos ψ cos φ sn ψ sn φ cos θ sn φ sn θ sn ψ sn φ sn θ cos ψ sn φ cos ψ sn φ sn ψ cos φ cos θ + cos φ sn θ sn ψ cos φ sn θ cos ψ cos θ sn ψ sn θ cos θ cos ψ.. R δx + R δy + R 3 δz z 0 )) }, v z ) = + x + y { R + R 3 a) x + R + R 3 b) y R δx + R δy + R 3 δz z 0 )) }, where a, b, and z /x + y ) / are suffcently small that ther squares are neglgble. For example, the maxmum devaton angle β max of a DIDSON system s 5, thus + tan β max ) / = Up to a frst order of approxmaton, we have, T = R R + R 3 a R + R 3 b δx + R δy +R 3 δz z 0 )) R R + R 3 a R + R 3 b δx + R δy +R 3 δz z 0 )) 0 0.6) Ths serves as a frst order approxmaton of the transform between two acoustc camera mages. Further approxmaton wll be studed n subsequent work by segmentng the mage nto local planes dependng on the levels of elevaton. The sx unknown parameters of the affne transform can be obtaned by matchng features n two mages. However, other parameters such as R j, a, b, or δx n 6) cannot be fgured out separately because those parameters are coupled and underconstraned. Consequently, under the above approxmaton, t s mpossble to reconstruct the precse egomoton of the acoustc camera merely based on mage regstraton parameters. III. METHODOLOGY The typcal four steps of mage regstraton are: feature detecton, feature matchng, transformaton estmaton, and mage resamplng and transformaton [0]. Feature detecton s the process of fndng objects such as corners, edges, lne ntersectons, etc. from mages, manually or automatcally. The features from the sensed mage are pared wth the correspondng features n the reference mage n the second step. In the thrd step, the transformaton s estmated based on the dsplacement vector of each feature. Once the mappng s establshed, the multple mages are combned to generate a fuson mage.

3 In our work, we have found that hgh curvature ponts can be useful as features of nterest n acoustc camera mages. The sum of squared dfference s used to measure the dssmlarty between two mages n the second step. Transform parameters are estmated va a random samplng based method. After the parameters of the affne transform are obtaned, all mages are combned by weghted average. A. Coordnate mappng and nhomogeneous nsonfcaton equalzaton In order to restore the spatal homogenety of the mage, a transformaton to the Cartesan coordnates has to be performed. Due to the fact that the feld of vew n the angular coordnate of dfferent sensors does not overlap, the resultng pxel sze n the Cartesan coordnates s not homogeneous. Therefore, nearest neghbor nterpolaton was appled to fll the gaps n the mage n the Cartesan coordnate system. Due to the acoustc acquston of mages whch was performed by nsonfyng the area wth a sngle source, an nhomogeneous ntensty profle s obtaned. Ths has to be corrected for effcent mosacng. Prevous work on separaton of llumnaton from reflectance has been based on Retnex theory [], []. Usng a homomorphc flterng method [3], one estmates the nsonfcaton of an mage, and reconstructs the mage under unform nsonfcaton. Modelng the nsonfcaton as a homomorphc flter, the mage produced by the sonar system can be wrtten as Ĩ = L M Î + η 7) where Ĩ s the -th observed mage, L the -th nsonfcaton operaton, M the -th down-samplng operaton, Î the ground truth mage under unform nsonfcaton, and η a Gaussan nose. The estmated nsonfcaton ntensty L s calculated by applyng ) a Gaussan flter to the orgnal mage, L = dag HĨ, where H s a Gaussan blurrng operaton. The estmated unform nsonfcaton mage s Ĩ u) = L L M Î + η) M Î + L η. 8) The sum of squared dfference between the -th and j-th unform nsonfcaton mages s SSD,j = Ĩ u) Ĩ u) j + L η L η 9) and can be used as a ds-smlarty measure. The second term n 9) s ndependent of the true mage, and may be regarded as a constant, provded the nose s unform. A regularzaton factor may be added to L to prevent erroneously excessve ntensty from the speckles n low nsonfcaton regons. B. Feature detecton and putatve matchng Feature detecton and matchng are computatonally demandng. A Gaussan pyramd has been proposed as a multscale approach for effcent feature detecton and matchng [4], [0]. A pxel n the polar coordnates mage corresponds to or several pxels n the mapped mage. For example, n an mage wth the range m, the number of pxels that correspond to a sngle pxel n the polar coordnates mage vared from to 8. Magnfed pxels result n jagged edges n the mapped mage. In our mages, feature detecton at the thrd level of the Gaussan pyramd reduces false detecton of corners at the jagged edges. Feature detecton and putatve matchng s ntalzed by translatonal dsplacement detecton. Translatonal dsplacement between the sensed mage and the reference mage s calculated by an exhaustve search on the fourth level of the Gaussan pyramd. Ths process drastcally reduces the area of exhaustve search. After translaton s estmated, hgh curvature ponts of the sensed mage are detected usng the Harrs corner detector [5]. The second moment matrx M s computed usng the followng relatonshp: M = e x x/σ s I) I) ), 0) where σ s s the scale factor of corners, and I s the gradent vector of the mage. The response after the Harrs corner detecton s, R = det M ktrm), ) where k s set to The local maxma of R correspond to corners. The corners are matched to the correspondng ponts n the reference mage by another exhaustve search on the thrd level of the Gaussan pyramd. C. Transform estmaton Image changes due to the sonar system movement are modeled by an affne transformaton as derved n the secton II. The affne transformaton descrbes the mage changes by yaw, small ptch and roll and translatonal movement of the sonar system. Ths s vald when multple objects are not present at the same range and angle, whch s the case wth the great majorty of mages n our dataset []. The detaled procedure of the algorthm s as follows: ) Feature ponts estmaton: Usng the Harrs corner detector, compute 50 nterest ponts n a preprocessed acoustc camera mage. ) Correspondng ponts search: For a square patch around each feature pont n the sensed mage, fnd the sub-pxel-wse dsplacement n the next mage, usng a cross-correlaton based matchng. ) Transform parameter estmaton: Repeat the followng )-3) for 000 samples.

4 ) Select 3 putatve matchng pars. ) Usng the matchng pars, estmate the parameters of the affne transform. 3) Fnd the nlers of the estmated transform, and repeat ) wth the nlers untl the estmated nlers are stablzed. v) Set a certan k percentle to defne a threshold n of feature ponts. Then, fnd the n pars of ponts that are closest to each other. The least mean squared error of the pars s used as the crteron. We use the crteron of least square error of k% of samples, where k s emprcally determned. It s smlar to the leastmedan of squares LMS) [6], but t dffers n that t can have a lower breakdown pont k nstead of 0.5 of LMS), and t uses the mean squared error nstead of the k-th percentle as the measure of error. It works well wth only a few feature pont pars wth hgh percentage of outlers. In addton, t yelds a measure of goodness of the transformaton, whch helps to decde whether to contnue mosacng or to stop, for example when the rsk of msmatch s hgh. D. Mosacng After regstraton, a mosac mage s constructed. Snce the nose s present regardless of the nsonfcaton condton, t can deterorate the mosac mage f not treated properly. For example, f we average well-nsonfed mages and poorly-nsonfed mages, the SNR wll become lower because nose wll be accumulated. A unformly nsonfed mage s, when addtve nose η of a standard devaton σ s added, Î + η σ ) and the SNR s smply /σ, and when t s nsonfed wth nhomogeneous nsonfcaton L, L Î + η σ 3) the SNR s σ dagl ). Mosacng va unform averagng can be descrbed as the followng relatonshp: Ĩ u) mosac = N the SNR of the mosac mage s N Ĩ, 4) = SNR u) mosac = σ N L. 5) Note that the SNR s a column vector of the same sze as ˆL because the nsonfcaton ntensty vares wthn the mage. A desrable stuaton s when poorly nsonfed regons receve lower weght n the averagng. Denote the weght of the -th mage by α, where α = I, and I s the dentty matrx. Then, the mosac mage produced va nhomogeneous weghtng s, of whch the SNR, SNR w) mosac s, Ĩ w) mosac = α Ĩ 6) SNR w) mosac α,..., α N ) = ) / α σ α L 7) ) / L σ. 8) Equalty holds n 8) when α k = L ) L k. Thus, the maxmum SNR of the mosac mage s acheved when the transformed mages are combned as follows: Ĩ w) mosac = L ) L I. 9) E. Maxmum-Lkelhood estmaton of equalzed mage Together wth the weghted mosac mage, the mage qualty can be enhanced by an addtonal process. Accordng to the nhomogeneous nsonfcaton model, the ntensty of a reflected sgnal vares dependng on the nsonfcaton condton. Gven the observed pxel values Ĩ, Ĩ,...Ĩ N under the nsonfcaton ntensty L, L,...L N, the estmated true reflectvty mage s Ĩ that maxmzes P Ĩ Ĩ, Ĩ,...Ĩ N ) = P Ĩ, Ĩ,...Ĩ N Ĩ)P Ĩ) P Ĩ, Ĩ,...Ĩ N ) 0) and, by approxmatng the Rcan nose dstrbuton by a Gaussan dstrbuton [7], we have the maxmum lkelhood mage Ĩ mle = arg max ln P Ĩ, Ĩ,...Ĩ N Ĩ) Ĩ = arg max { Ĩ σ L L Ĩ Ĩ) +... ) } +L N L N ĨN Ĩ where the denomnator of 0) s neglected. By combnng the exponents, fnally, where Ĩ mle = arg max ) Ĩ0 Ĩ σ0 Ĩ ) σ 0 = σ L, ) Ĩ 0 = L ) L Ĩ. 3)

5 Ths result s useful snce we can choose an approprate pror P Ĩ) to get the best a posteror estmaton of the equalzed mage. Note that the maxmum-lkelhood estmaton result has the same weght as n 9), and the only dfference s that the sum of the nsonfcaton ntensty n 9) s replaced by the square sum n 3). Therefore, ths maxmum lkelhood estmaton mage has the same maxmum SNR property as the weghted mosac mage n 9). F. Estmaton of the Pror from Pont Spread Functon A more detaled procedure of mage formaton from an magng devce s typcally modelled by the followng equaton: Ĩ = L M P Î + η where Î represents the true or hgh-resoluton mage, P the -th pont spread functon operaton, M the down-samplng operaton, L the llumnaton or nsonfcaton for sonar systems), η the addtve nose, and Ĩ the -th observed mage. The process pont spread functon PSF) and downsamplng operaton fuse multple pxels n the hgh resoluton mage nto one pxel n the low resoluton mage. Usually, the fuson s performed wth dfferent spatal weghts gven to the hgh resoluton pxels, whch s n most of the cases modeled by Gaussan dstrbutons. The fuson process can be represented as the followng: θ w = θ 4)... ) w =, 5) where θ s a low resoluton pxel value, w the weght vector, θ a vector of the pxels n the hgh resoluton mage that correspond to the pxel n the low resoluton mage. On the other hand, we have another constrant to estmate the probablty densty functon θ, whch s that a pxel value s not ndependent of ts adjacent pxel values. A Gaussan Markov Random Feld s one of the popular models for ths property, whch gves us the followng probablty densty functon of θ: ) P θ) = N exp θ D D)θ, 6) where D s an operaton of frst dervatve, and N a normalzaton constant. Combnng those two constrants, we obtan the pror, and the fnal maxmum a posteror MAP) mage can be obtaned: ) Ĩ map = L P L P Ĩ, 7) where P s the pont spread functon operated onto the ground truth mage. IV. RESULTS The algorthm was tested on a boat wreckage sequence. A DIDSON system [] scanned a shpwreck at approxmately 30 meters depth for 85 seconds and took 446 frames of mages. About 40 frames among them show the vessel from head to stern, and another 40 frames show t from stern to head. We appled the algorthm to those two sub-sequences, and bult two mosac mages. Fg. depcts two consecutve acoustc mages together wth a set of matched crcle) and non-matched trangle) feature ponts. These matched feature ponts n the reference mage, whch were found usng the cross-correlaton of patches around the feature ponts n the sensed mage, are used to estmate the geometrc transformaton between the two mages. Fg.. Matched crcle) and non-matched trangle) feature ponts whch were obtaned from the thrd level of the Gaussan pyramd usng the Harrs corner detector. Features from a sensed mage are pared wth correspondng ponts n the reference mage. A 5 5 patch around each feature pont n the sensed mage s matched wth the same szed patch from the correspondng area n the reference mage. The outlers features wth weaker matchng) are defned by those pars wth hgher matchng error after the estmated transformaton. Cross-correlaton was found to be more robust than a more conventonal approach n whch features are ndependently found and matched between the two mages. Ths s a consequence of the hgh nose n the mage and the fact that the exact locaton of the feature s not so well defned. Fg. 3 represents the man result of the paper,.e., a mosac mage of multple acoustc mages. The mosaced mage contans nformaton whch spans multple frames, each frame correspondng to a small porton of the nsonfed object. The combnaton mages whch have been transformed to be n the same coordnate system provde subpxel mage resoluton enhancement. Fg. 4a) shows a frame of another shpwreck target before mosacng. The resoluton enhancement follows from the fact that one pxel n the orgnal polar coordnate system s mapped to multple pxels wth the same ntensty n the Cartesan coordnate system. Dfferent frames lead to partal overlap of these The data has been provded by E. O. Belcher from Appled Physcs Laboratory, Unversty of Washngton under ONR support.

6 multple pxels, so that after averagng a subpxel resoluton s acheved See Fg. 4a)). Averagng of dfferent acoustc mages after brngng them to the same coordnate system same vewpont) leads to the classcal effect of denosng. Ths s clearly seen n Fg. 3 on the whole target, and n partcular n the comparson of a detaled structure of the other target n Fg. 4. The top two panels of Fg. 3 depct a mosaced mage from the same sequence of acoustc mages. In panel b), the nsonfcaton profle was utlzed durng averagng. Panel c) and d) represent the same target from a dfferent acoustc mage sequence wth panel d) utlzng the nsonfcaton profle durng averagng. a) b) V. CONCLUSION Acoustc camera technology s becomng essental for underwater exploraton, and has the potental for the applcaton to other areas for vsual nvestgatons. The acoustc camera, wth ts specfc sensor desgn, poses some challenges n terms of mage resoluton, nose removal and area coverage. In ths paper, we have presented a complete algorthm to acheve mage mosacng, denosng and resoluton enhancement from a sequence of acoustc camera mages. We descrbed the steps that were requred to acheve ths mosacng. Ths ncluded modelng the specfc geometry of acoustc camera mages whch sharply dffers from pnhole camera geometry. The dfferent geometry, and n partcular, the fact that the mages are acqured n a polar coordnate system, complcates the search and matchng of feature ponts n consecutve mages. Moreover, n ths partcular geometry, pxels n the polar coordnate system are mapped to a collecton of pxels wth the same ntensty n the Cartesan coordnate system. Snce consecutve mages were taken from dfferent vewponts, a subpxel enhancement effect was acheved n the process of averagng n addton to the denosng effect. We have presented a novel method n whch features extracted by a certan algorthm are locally matched to the reference mage va cross-correlaton. Ths method was found to be more robust than a conventonal approach n whch features are ndependently found and matched between two mages. In partcular, ths s more pronounced when the number of pxels avalable for feature comparson s lmted. The dfferent geometry, and n partcular the fact that the mages are acqured n a polar coordnate system complcates the search and matchng of feature ponts n consecutve mages. Moreover, n ths partcular geometry, pxels n the polar coordnate system are mapped to a collecton of pxels wth the same ntensty n the Cartesan coordnate system. Snce consecutve mages were taken from dfferent vewponts, a subpxel enhancement effect was acheved n the process of c) d) Fg. 3. Demonstraton of weghted averagng effect: Mosacng was performed wth 38 mages whch were averaged after the correspondng moton compensaton transformaton was appled to each of them. Panel a) demonstrates the unform average of the whole mages, whle panel b) demonstrates the weghted average, n whch nsonfcaton profle was utlzed durng averagng. See secton III. D for detals.) Panel c) and d) depct the same target from a dfferent mage sequence. averagng n addton to the denosng effect. We have presented a method n whch features are found n a one mage, and are then locally matched to the another mage va crosscorrelaton. ACKNOWLEDGMENT Ths work was partly supported by ONR grant N C-096. We thank E. O. Belcher for answerng all our questons about the data. We would lke to send Professor Leon N Cooper and other members of IBNS for provdng valuable comments.

7 a) Fg. 4. Resoluton enhancement by weghted averagng mages. a) A sngle frame mage. b) A mosac mage of 7 consecutve frames followed by a geometrc transformaton. See Secton III. F.) b) REFERENCES [] E. O. Belcher, B. Matsuyama, and G. M. Trmble, Object dentfcaton wth acoustc lenses, n Proceedngs of Oceans 0 MTS/IEEE, 00, pp. 6. [] R. F. Wagner, S. W. Smth, J. M. Sandrk, and H. Lopez, Statstcs of speckle n ultrasound b-scans, IEEE Transactons on Soncs and Ultrasoncs, vol. 30, no. 3, pp , 983. [3] R. L. Marks, S. M. Rock, and M. J. Lee, Real-tme vdeo mosackng of the ocean floor, IEEE Journal of Oceanc Engneerng, vol. 0, no. 3, pp. 9 4, 995. [4] Y. Rzhanov, L. M. Lnnet, and R. Forbes, Underwater vdeo mosacng for seabed mappng, n Internatonal Conference on Image processng, 000, vol., pp [5] U. Castellan, A. Fusello, and V. Murno, Regstraton of multple acoustc range vews for underwater scene reconstructon, Computer Vson and Image Understandng, vol. 87, pp , 00. [6] R. García, X. Cufí, and L. Pacheco, Image mosackng for estmatng the moton of an underwater vehcle, n 5th IFAC Conference on Manoeuvreng and Control of Marne Craft, 000. [7] S. Negahdarpour, X. Xu, and A. Khamene, A vson system for realtme postonng, navgaton and vdeo mosacng of sea floor magery n the applcaton of rovs/auvs, n IEEE Workshop on Applcatons of Computer Vson, 998, pp [8] D. Capel and A. Zsserman, Computer vson appled to superresoluton, IEEE Sgnal Processng Magazne, vol. 0, no. 3, pp. 7 86, 003. [9] A. Smolc and T. Wegand, Hgh-resoluton vdeo mosacng, n Internatonal Conference on Image Processng, 00, vol. 3, pp [0] B. Ztová and J. Flusser, Image regstraton methods: a survey, Image and Vson Computng, vol., pp , 003. [] E. H. Land, Recent advances n the retnex theory and some mplcatons for cortcal computatons: Color vson and the natural mage, PNAS, vol. 80, pp , 983. [] E. H. Land and J. J. McCann, Lghtness and the retnex theory, Journal of the Optcal Socety of Amerca, vol. 6, pp., 97. [3] E. H. Land, An alternatve technque for the computaton of the desgnator n the retnex theory of color vson, PNAS, vol. 83, pp , 986. [4] D. A. Forsyth and J. Ponce, Computer Vson: A Modern Approach, Pearson Educaton, Inc., Upper Saddle Rver, NJ, 003. [5] C. J. Harrs and M. Stephens, A combned corner and edge detector, n Proceedngs on 4th Alvey Vson Conference, 988, pp [6] P. J. Rousseeuw, Least medan of square regresson, Journal of the Amercan Statstcal Assocaton, vol. 79, pp , 984. [7] J. Sjbers, J. den Dekker, P. Scheunders, and Drk Van Dyck, Maxmumlkelhood estmaton of rcan dstrbuton parameters, IEEE Transactons on Medcal Imagng, vol. 8, no. 3, pp , 998.